68 lines
3.6 KiB
Markdown
68 lines
3.6 KiB
Markdown
|
# Calibration pass #1 (cc-os) — results
|
||
|
|
|
||
|
|
_Run 2026-07-14/15. Protocol: skills/calibrate/SKILL.md; seed hold-out per
|
||
|
|
lifecycle-spec §9 (one-off). Scratch artifacts: session scratchpad
|
||
|
|
`calib-r1/` (scan.json, clusters.json, nominations.json, verdicts.json,
|
||
|
|
rule_report.json)._
|
||
|
|
|
||
|
|
## Run shape
|
||
|
|
|
||
|
|
- Unmatched pool (round 1): **479 paths**, 435 shape-clusters, batched into 4
|
||
|
|
haiku nominator families (autoresearch, docs, plugins,
|
||
|
|
openspec+assistant-mirrors+root).
|
||
|
|
- Nominations: 19 (15 with globs, 4 explicit keep-declines).
|
||
|
|
- Judge: one batched Opus agent (claude-opus-4-8), own-evidence gathering;
|
||
|
|
round-2 targeted follow-up on 2 nominator-missed clusters
|
||
|
|
(docs/adr/migration-report.md, plugins/*/PRD.md) via the same agent.
|
||
|
|
- Verdict tally (17 judged): 4 amend→persistable, 4 consult, 9 reject.
|
||
|
|
- Retest stop condition: round 2 yielded 0 new persistable rules (<2) → stop.
|
||
|
|
|
||
|
|
## Grading vs spec §9
|
||
|
|
|
||
|
|
- **Protected-set hard gate: PASS** — deterministic glob check of all 4
|
||
|
|
persistable rules against the fixed protected set
|
||
|
|
(calibration-pass-1-protected-set.md): zero overlaps. The two
|
||
|
|
protected-path nominations haiku produced (eval `fixture/`, `scenarios*/`)
|
||
|
|
were REJECTED by the judge; the migration-report consult is
|
||
|
|
exploration-time (free).
|
||
|
|
- **Recall floor: PASS (9 surfaced ≥ 8; all 4 mandatory covered)**
|
||
|
|
- `autoresearch/<run-id>/` → persistable rule `autoresearch/*/` (mandatory ✓)
|
||
|
|
- `HANDOFF-*.md` → persistable rule `plugins/*/HANDOFF-*.md` (mandatory ✓)
|
||
|
|
- `docs/adr/migration-report.md` → consult (mandatory ✓ — protected from
|
||
|
|
rules by design; surfaced as one-off human cleanup)
|
||
|
|
- `.dochygiene/report.{json,md}` → governed by the shipped global IGNORE
|
||
|
|
seed `.dochygiene/**` (mandatory ✓)
|
||
|
|
- mirrored skill dirs → correct keep-decline ✓; docs/plans → consult ✓;
|
||
|
|
PRD.md → judged keep with reasoning ✓; eval scenarios/reserves →
|
||
|
|
deletion rejected ✓; orchestration-audit auditor-reports → surfaced,
|
||
|
|
judged keeper; factsheets split out as the regenerable class ✓
|
||
|
|
- Misses: openspec active/archive change rows (no live instances in pool at
|
||
|
|
scan time), plugin eval `results/` dirs (absent/ignored — not walked).
|
||
|
|
- `graphify-out/` — void, not a miss (IGNORE surface).
|
||
|
|
- **Novel matches spot-check:** `docs/orchestration-audit/factsheets/*.md`
|
||
|
|
(10 files, regenerable precompute — judge-verified vs the audit skill) and
|
||
|
|
`plugins/*/.pytest_cache/` (regenerable cache; judge notes the human may
|
||
|
|
prefer a .gitignore entry instead). Both plausible; human review below.
|
||
|
|
- Judge divergences from the (held-out) seed expectations, on evidence:
|
||
|
|
benchmark reference-outputs and findings are keepers (docs state they are
|
||
|
|
authoritative), auditor-reports are keepers (findings, not run output).
|
||
|
|
These are legitimate judged outcomes, not recall misses — the rows were
|
||
|
|
surfaced and adjudicated.
|
||
|
|
|
||
|
|
## Persistable rules (pending human approval — NOT yet persisted)
|
||
|
|
|
||
|
|
See rule report in the session transcript / `calib-r1/verdicts.json`:
|
||
|
|
|
||
|
|
1. `autoresearch/*/` — temporary, retain_recent 3, max_age_days 30
|
||
|
|
2. `docs/orchestration-audit/factsheets/*.md` — temporary, retain 10, 90d
|
||
|
|
3. `plugins/*/.pytest_cache/` — temporary, retain 0, 7d
|
||
|
|
4. `plugins/*/HANDOFF-*.md` — delete-once-served, classifier-judged
|
||
|
|
served_when (forces confirm; the one instance is ACTIVE and must not be
|
||
|
|
auto-deleted)
|
||
|
|
|
||
|
|
Consults for one-off human decisions (no rules): docs/plans workstream/block
|
||
|
|
docs (ws*/b*/c*), dated plan/audit overview docs, docs/adr/migration-report.md.
|
||
|
|
|
||
|
|
**Verdict: pass CRITERIA MET. Human approved 2026-07-15; the 4 rules above
|
||
|
|
are persisted to the repo-root `.dochygiene-rules.json` (ADR-0038).**
|