cc-os/plugins/os-doc-hygiene/openspec/specs/calibrate/spec.md

# Spec: calibrate

## Purpose

Defines the `:calibrate` skill, which proposes new lifecycle rulebook rules by
clustering the pool of files the rulebook currently leaves unmatched, having a
cheap model nominate generalizable glob patterns, and having a strong model
judge and report those nominations to a human before any persistence.

## Requirements

### Requirement: Calibrate Clusters and Samples Over Unmatched Files

The `:calibrate` skill SHALL run over the unmatched-files pool (unmatched =
unmanaged, per `lifecycle-rulebook`) as its candidate pool. It SHALL cluster
unmatched paths by shape before nominating any rule, so that a proposed rule
is authored against a cluster of similar paths rather than a single file.

#### Scenario: Calibrate operates only on the unmatched pool

- **WHEN** `:calibrate` runs
- **THEN** its candidate pool is exactly the set of files the rulebook currently leaves unmatched

#### Scenario: Rules are proposed against clusters, not single files

- **WHEN** `:calibrate` nominates a candidate rule
- **THEN** the nomination is derived from a cluster of similar unmatched paths, not from one instance in isolation

### Requirement: Cheap-Model Nomination Produces Patterns, Never Exact-Instance Globs

For each sampled cluster, `:calibrate` SHALL dispatch a haiku subagent
constrained to nominate a bare glob pattern plus a candidate lifetime. The
haiku nomination SHALL be constrained to produce generalizable patterns; it
SHALL NOT be accepted as final if it hardcodes an identifier unique to a
single instance (a run-id, hash, or bare timestamp).

#### Scenario: Haiku nominates a glob and lifetime per cluster

- **WHEN** a cluster of unmatched paths is sampled
- **THEN** the haiku subagent returns a bare glob pattern and a candidate lifetime for that cluster

#### Scenario: Instance-unique nominations are not accepted as final

- **WHEN** a haiku nomination's glob hardcodes a run-id, hash, or bare timestamp unique to one file
- **THEN** it is not persisted as-is; it must be caught and generalized before the strong-model judgment step, per the rule-quality tests below

### Requirement: Strong-Model Batched Judgment with Confirm/Reject/Amend/Consult

`:calibrate` SHALL dispatch one batched strong-model (Opus or Fable) judge
subagent that gathers its own evidence (re-reading matched paths and
checking near-miss boundaries) and authors final rule entries. The judge
SHALL return one of four verdicts per nominated rule: `confirm`, `reject`,
`amend`, or `consult`. `consult` SHALL be mandatory whenever an artifact's
purpose is unclear (i.e., the judge cannot determine whether the artifact is
regenerable or must be retained).

#### Scenario: Judge gathers its own evidence rather than trusting the nomination

- **WHEN** the strong-model judge evaluates a haiku nomination
- **THEN** it independently re-reads matched paths and checks near-miss boundaries rather than accepting the nomination's claims at face value

#### Scenario: Four verdicts are the only possible outcomes

- **WHEN** the judge evaluates a nominated rule
- **THEN** its verdict is exactly one of `confirm`, `reject`, `amend`, or `consult`

#### Scenario: Consult is mandatory when purpose is unclear

- **WHEN** the judge cannot determine whether a clustered artifact type is regenerable or must be retained
- **THEN** the verdict is `consult`, never `confirm` or `reject`

### Requirement: Rule Report to the Human Before Persistence

Before any proposed rule is persisted, `:calibrate` SHALL present a rule
report to the human containing, per proposed rule: the glob verbatim exactly
as it would be persisted; every path it currently matches (or a capped
sample plus a total count); the near-miss boundary — paths that do NOT match
despite looking similar; the lifetime and behavior tier (auto vs confirm);
and a plain-language explanation of what the artifact is and why it is
clutter. No rule SHALL be persisted before this report has been shown.

#### Scenario: The report shows the exact persisted glob

- **WHEN** a rule report is generated for a proposed rule
- **THEN** the glob shown is character-for-character identical to what would be written to the rulebook file

#### Scenario: The report shows current matches with a capped sample

- **WHEN** a proposed rule matches more paths than the display cap
- **THEN** the report shows a capped sample of matched paths plus the total count of all matches

#### Scenario: The report shows the near-miss boundary

- **WHEN** a proposed rule's glob narrowly excludes similar-looking paths
- **THEN** the report explicitly lists those near-miss non-matching paths, so a boundary bug (e.g. a glob silently missing a sibling path) is visible before persistence

#### Scenario: No rule is persisted before the report is shown

- **WHEN** `:calibrate` has generated proposed rules
- **THEN** it does not write any rule to a rulebook file until the human has seen the rule report for it

### Requirement: Persistence Rules by Scope

Project-rulebook writes SHALL land on judge confirmation once the human has
reviewed the rule report. Global-rulebook writes (writing into
`plugins/os-doc-hygiene/rulebook.json`) SHALL additionally require explicit
human gating, distinct from project-rule confirmation, since it is a
cross-repo write into cc-os. Rule removals SHALL be HITL-only in all cases,
with recorded reasoning, regardless of scope. Every settled verdict SHALL
persist: judge `keep` verdicts become ordinary `lifetime: keep` rules in
`rules` (including exact-path singletons, per the keep-tier relaxation);
human declines at the rule report persist as `rejected` entries with
`rejected_by: "human"`; open `consult` verdicts persist to
`nominations.consults`, deduped by glob at write time. All persistence SHALL
go through the canonical rules-file writer.

#### Scenario: Project rule persists on judge confirmation plus report review

- **WHEN** the judge verdict is `confirm` for a project-scoped rule and the human has reviewed its rule report
- **THEN** the rule is written to the project's `.dochygiene-rules.json`

#### Scenario: Global rulebook writes require an additional explicit gate

- **WHEN** a proposed rule would be written to the global `rulebook.json`
- **THEN** a distinct human confirmation for the cross-repo write is required, beyond the project-rule confirmation step

#### Scenario: Rule removal is always HITL-only

- **WHEN** any rule (project or global) is proposed for removal
- **THEN** the removal happens only via explicit human instruction, with the reasoning recorded, never as an automatic side effect of a calibration pass

#### Scenario: A keep verdict persists as a plain keep rule

- **WHEN** the judge's settled verdict for a cluster is that the artifacts must be retained
- **THEN** an ordinary `lifetime: keep` rule is persisted to `rules` — removing the cluster from future calibrate pools and protecting future matches — even when the glob is an exact-path singleton

#### Scenario: A human decline persists as a rejection

- **WHEN** the human declines a judge-confirmed rule at the rule report
- **THEN** a `rejected` entry with `rejected_by: "human"` is written, so a later haiku round cannot re-nominate the identical glob+lifetime without the judge knowing

### Requirement: Retest Loop with Stop Conditions and Hard Cap

`:calibrate` SHALL re-run its clustering pass against the shrunk unmatched
pool after each round of persisted rules. It SHALL stop when a round yields
fewer than 2 new rules OR shrinks the unmatched pool by less than 10%. It
SHALL hard-cap at 3 rounds regardless of shrink rate.

#### Scenario: Stops on fewer than 2 new rules

- **WHEN** a retest round yields only 1 new confirmed rule
- **THEN** the retest loop stops after that round

#### Scenario: Stops on less than 10% shrink

- **WHEN** a retest round shrinks the unmatched pool by less than 10%
- **THEN** the retest loop stops after that round, even if 2 or more rules were confirmed

#### Scenario: Hard cap of 3 rounds regardless of shrink

- **WHEN** three retest rounds have run and each still meets the continuation criteria (≥2 new rules and ≥10% shrink)
- **THEN** the retest loop stops after the third round regardless

### Requirement: Seed Intake at Judge Intake

The clutter-inventory seed candidates SHALL enter the calibration protocol at
judge intake. Full seed intake SHALL apply to every calibration run after
calibration pass #1; pass #1 uses the one-off seed hold-out described in the
`Calibration Pass Validation Criteria` requirement below.

#### Scenario: Seed candidates feed the judge step

- **WHEN** a calibration run (other than pass #1) begins
- **THEN** the clutter-inventory seed candidates are included as judge-intake evidence

### Requirement: Rule-Quality Test — Class Never Path

A proposed rule's glob SHALL name a recurring class of artifact, never an
identifier unique to a single instance. A glob that hardcodes a name
recurring by convention (e.g. `PRD.md`, `HANDOFF-*.md`,
`migration-report.md`) is acceptable. A glob that hardcodes a run-id, hash,
or bare timestamp unique to one instance is not acceptable. A rule that
currently matches only one file is acceptable; a rule that can, by
construction, only ever match one file is a failed generalization and SHALL
be flagged loudly rather than silently persisted. **Keep-tier relaxation:**
exact-path/instance globs ARE acceptable for `lifetime: keep` entries only —
this test exists to prevent bad deletion rules, and a singleton keep merely
protects; instance globs remain forbidden for `temporary` and
`delete-once-served`. When the near-miss boundary check reveals sibling
artifacts a glob misses, the fix SHALL be to enumerate the conventional
prefixes as separate rule entries, never to widen the glob to the containing
directory; container-claiming globs are justified ONLY when the directory is
wholly machine-owned (e.g. `plugins/*/.pytest_cache/`).

#### Scenario: A convention-recurring name is acceptable

- **WHEN** a proposed rule's glob is `HANDOFF-*.md`
- **THEN** it passes the class-never-path test, since `HANDOFF-*` is a recurring naming convention, not a single instance

#### Scenario: An instance-unique identifier fails the test

- **WHEN** a proposed rule's glob hardcodes a specific run-id or hash string that can only ever identify one artifact
- **THEN** the rule fails the class-never-path test and is flagged loudly, not silently persisted

#### Scenario: One current match is fine; one-EVER match is not

- **WHEN** a proposed rule currently matches exactly one file
- **THEN** it is acceptable if the glob's structure could match future similarly-named files; it is flagged as a failed generalization if the glob's structure can never match any file but the one it names today

#### Scenario: A singleton keep passes under the keep-tier relaxation

- **WHEN** a proposed rule is `docs/research/clutter-pattern-inventory.md -> keep`
- **THEN** it passes despite being an exact-path instance glob, because the keep tier only protects; the same glob with lifetime `temporary` or `delete-once-served` fails

#### Scenario: Missed siblings are enumerated, never widened to the container

- **WHEN** the boundary check shows `autoresearch/classic-*/` misses sibling runs under `autoresearch/improve-*/`
- **THEN** the fix is a second rule entry `autoresearch/improve-*/` (same cluster), never a widening to `autoresearch/*/` or `autoresearch/**`, so keep-worthy content can still live in the container without a counter-rule

### Requirement: Rule-Quality Tie-Breaker — Prefer the Narrower Glob

When choosing between candidate globs of differing breadth for the same
cluster, `:calibrate` SHALL prefer the narrower glob. Too-narrow failure is
self-healing (clutter is merely left for a later round to catch); too-broad
failure is dangerous (a keeper file may be deleted and is not
self-healing).

#### Scenario: Narrower glob is chosen when both would satisfy the cluster

- **WHEN** two candidate globs both cover the sampled cluster, one narrower and one broader
- **THEN** the narrower glob is chosen

#### Scenario: Rationale is evidence quality, not readability alone

- **WHEN** justifying the narrower-glob preference
- **THEN** the reasoning cited is that too-narrow fails safe (self-healing) while too-broad fails dangerous (not self-healing), not merely stylistic preference

### Requirement: Calibration Pass Validation Criteria

A calibration pass SHALL be judged against: a precision hard gate (the pass
FAILS if any persisted rule's glob matches a protected path, regardless of
behavior tier — exploration-time `consult` verdicts on protected paths are
free and do not fail the pass); a recall floor of 8 of the 10 rows of the
project's clutter inventory, with 4 specific rows mandatory (missing any
mandatory row fails the pass); a one-off seed hold-out for calibration pass
#1 only (the sealed answer key is withheld from judge intake for pass #1;
every later run uses full seed intake); treatment of IGNORE-surface paths as
void, not a miss, against the recall floor; and the requirement that a
do-nothing pass cannot pass (the recall floor makes the pass falsifiable in
the finding direction).

#### Scenario: A rule matching a protected path fails the pass

- **WHEN** any persisted rule's glob matches a path in the fixed protected set
- **THEN** the calibration pass fails, regardless of whether that rule's tier was auto or confirm

#### Scenario: Consult verdicts on protected paths do not fail the pass

- **WHEN** the judge issues a `consult` verdict during exploration for a candidate touching a protected path, and that candidate is not persisted
- **THEN** the pass is not failed by that consult verdict

#### Scenario: Recall floor requires 8 of 10 with 4 mandatory

- **WHEN** a calibration pass is graded against the clutter inventory
- **THEN** it must recall at least 8 of the 10 inventory rows, and all 4 mandatory rows must be among them, or the pass fails

#### Scenario: Pass #1 seed hold-out is a one-off

- **WHEN** calibration pass #1 runs
- **THEN** the sealed answer key (the project's clutter-inventory rows) is withheld from judge intake; every subsequent calibration run instead uses full seed intake

#### Scenario: IGNORE-surface rows are void, not a miss

- **WHEN** grading recall against the clutter inventory and a row corresponds to an IGNORE-surface path
- **THEN** that row is excluded from the recall calculation entirely (void), not counted as a miss

#### Scenario: A do-nothing pass cannot pass

- **WHEN** a calibration pass persists zero rules
- **THEN** it fails the recall floor and therefore cannot pass

### Requirement: Deterministic Nomination Intake Filter

Between cheap-model nomination and strong-model judgment, `:calibrate` SHALL
run a deterministic `NominationIntakeFilter` (in `calibrate_helpers.py`, no
model — invariant #6). A nomination whose glob+lifetime exactly equals a
`rejected` entry SHALL be dropped before the judge and logged in the run
summary. Surviving nominations SHALL be annotated with every related
rejection, where related means the two globs' match sets intersect on the
current shortlist (deterministic, computed from the scan). The annotations
plus all open consults SHALL enter the judge prompt as its "Nominations
memory" input section; related rejections are context for the judge, never a
veto.

#### Scenario: Exact glob+lifetime repeat is dropped before the judge

- **WHEN** haiku nominates `docs/research/** -> temporary` and `nominations.rejected` contains an entry with glob `docs/research/**` and lifetime `temporary`
- **THEN** the nomination is dropped at intake, never reaches the judge, and the drop is logged in the run summary

#### Scenario: A variant flows through annotated, not blocked

- **WHEN** haiku nominates `docs/research/drafts/** -> temporary` and `nominations.rejected` contains `docs/research/** -> temporary`, and the two globs' match sets intersect on the current shortlist
- **THEN** the nomination proceeds to the judge carrying the related rejection (its why and consider_instead) as context, and the judge may still confirm it

#### Scenario: Open consults always reach the judge prompt

- **WHEN** `nominations.consults` is non-empty at intake time
- **THEN** every open consult is included in the judge prompt's "Nominations memory" section, regardless of what haiku nominated this round

### Requirement: Consult Persistence and Resurfacing

Open `consult` verdicts SHALL persist to `nominations.consults` (deduped by
glob at write time) rather than dying with the run. Consults SHALL resurface
in `:calibrate` only — `:check` and `:clean` are unchanged — appearing in the
judge prompt and as an "Open consults" section of the rule report. A consult
SHALL exit in exactly one of three ways: (a) a human answer settles the
purpose — a normal rule is persisted and the consult entry deleted; (b) the
human deems it not rule-worthy — the entry is rewritten into `rejected` with
`rejected_by: "human"` and the human's why; (c) the human defers — the entry
stays and resurfaces next run. New rejections and consults SHALL appear in
the rule report but are not individually gated — they are memory, not
deletion authority.

#### Scenario: A consult survives the run and resurfaces

- **WHEN** a judge verdict is `consult` and the run ends without a human answer
- **THEN** the consult is written to `nominations.consults`, and the next `:calibrate` run surfaces it in both the judge prompt and the report's "Open consults" section

#### Scenario: An answered consult becomes a rule and disappears

- **WHEN** the human answers an open consult in a way that settles the artifact's purpose
- **THEN** a normal rule is persisted through the standard report flow and the consult entry is deleted in the same write

#### Scenario: A declined consult becomes a human rejection

- **WHEN** the human answers that an open consult's artifact class is not rule-worthy
- **THEN** the consult entry is rewritten into `nominations.rejected` with `rejected_by: "human"` and the stated reason

#### Scenario: Consults never surface outside calibrate

- **WHEN** `:check` or `:clean` runs against a project with open consults
- **THEN** their behavior is unchanged — consults neither appear in output nor affect any classification