# Spec: calibrate

## ADDED Requirements

### Requirement: Calibrate Clusters and Samples Over Unmatched Files

The `:calibrate` skill SHALL run over the unmatched-files pool (unmatched =
unmanaged, per `lifecycle-rulebook`) as its candidate pool. It SHALL cluster
unmatched paths by shape before nominating any rule, so that a proposed rule
is authored against a cluster of similar paths rather than a single file.

#### Scenario: Calibrate operates only on the unmatched pool

- **WHEN** `:calibrate` runs
- **THEN** its candidate pool is exactly the set of files the rulebook currently leaves unmatched

#### Scenario: Rules are proposed against clusters, not single files

- **WHEN** `:calibrate` nominates a candidate rule
- **THEN** the nomination is derived from a cluster of similar unmatched paths, not from one instance in isolation

### Requirement: Cheap-Model Nomination Produces Patterns, Never Exact-Instance Globs

For each sampled cluster, `:calibrate` SHALL dispatch a haiku subagent
constrained to nominate a bare glob pattern plus a candidate lifetime. The
haiku nomination SHALL be constrained to produce generalizable patterns; it
SHALL NOT be accepted as final if it hardcodes an identifier unique to a
single instance (a run-id, hash, or bare timestamp).

#### Scenario: Haiku nominates a glob and lifetime per cluster

- **WHEN** a cluster of unmatched paths is sampled
- **THEN** the haiku subagent returns a bare glob pattern and a candidate lifetime for that cluster

#### Scenario: Instance-unique nominations are not accepted as final

- **WHEN** a haiku nomination's glob hardcodes a run-id, hash, or bare timestamp unique to one file
- **THEN** it is not persisted as-is; it must be caught and generalized before the strong-model judgment step, per the rule-quality tests below

### Requirement: Strong-Model Batched Judgment with Confirm/Reject/Amend/Consult

`:calibrate` SHALL dispatch one batched strong-model (Opus or Fable) judge
subagent that gathers its own evidence (re-reading matched paths and
checking near-miss boundaries) and authors final rule entries. The judge
SHALL return one of four verdicts per nominated rule: `confirm`, `reject`,
`amend`, or `consult`. `consult` SHALL be mandatory whenever an artifact's
purpose is unclear (i.e., the judge cannot determine whether the artifact is
regenerable or must be retained).

#### Scenario: Judge gathers its own evidence rather than trusting the nomination

- **WHEN** the strong-model judge evaluates a haiku nomination
- **THEN** it independently re-reads matched paths and checks near-miss boundaries rather than accepting the nomination's claims at face value

#### Scenario: Four verdicts are the only possible outcomes

- **WHEN** the judge evaluates a nominated rule
- **THEN** its verdict is exactly one of `confirm`, `reject`, `amend`, or `consult`

#### Scenario: Consult is mandatory when purpose is unclear

- **WHEN** the judge cannot determine whether a clustered artifact type is regenerable or must be retained
- **THEN** the verdict is `consult`, never `confirm` or `reject`

### Requirement: Rule Report to the Human Before Persistence

Before any proposed rule is persisted, `:calibrate` SHALL present a rule
report to the human containing, per proposed rule: the glob verbatim exactly
as it would be persisted; every path it currently matches (or a capped
sample plus a total count); the near-miss boundary — paths that do NOT match
despite looking similar; the lifetime and behavior tier (auto vs confirm);
and a plain-language explanation of what the artifact is and why it is
clutter. No rule SHALL be persisted before this report has been shown.

#### Scenario: The report shows the exact persisted glob

- **WHEN** a rule report is generated for a proposed rule
- **THEN** the glob shown is character-for-character identical to what would be written to the rulebook file

#### Scenario: The report shows current matches with a capped sample

- **WHEN** a proposed rule matches more paths than the display cap
- **THEN** the report shows a capped sample of matched paths plus the total count of all matches

#### Scenario: The report shows the near-miss boundary

- **WHEN** a proposed rule's glob narrowly excludes similar-looking paths
- **THEN** the report explicitly lists those near-miss non-matching paths, so a boundary bug (e.g. a glob silently missing a sibling path) is visible before persistence

#### Scenario: No rule is persisted before the report is shown

- **WHEN** `:calibrate` has generated proposed rules
- **THEN** it does not write any rule to a rulebook file until the human has seen the rule report for it

### Requirement: Persistence Rules by Scope

Project-rulebook writes SHALL land on judge confirmation once the human has
reviewed the rule report. Global-rulebook writes (writing into
`plugins/os-doc-hygiene/rulebook.json`) SHALL additionally require explicit
human gating, distinct from project-rule confirmation, since it is a
cross-repo write into cc-os. Rule removals SHALL be HITL-only in all cases,
with recorded reasoning, regardless of scope.

#### Scenario: Project rule persists on judge confirmation plus report review

- **WHEN** the judge verdict is `confirm` for a project-scoped rule and the human has reviewed its rule report
- **THEN** the rule is written to the project's `.dochygiene-rules.json`

#### Scenario: Global rulebook writes require an additional explicit gate

- **WHEN** a proposed rule would be written to the global `rulebook.json`
- **THEN** a distinct human confirmation for the cross-repo write is required, beyond the project-rule confirmation step

#### Scenario: Rule removal is always HITL-only

- **WHEN** any rule (project or global) is proposed for removal
- **THEN** the removal happens only via explicit human instruction, with the reasoning recorded, never as an automatic side effect of a calibration pass

### Requirement: Retest Loop with Stop Conditions and Hard Cap

`:calibrate` SHALL re-run its clustering pass against the shrunk unmatched
pool after each round of persisted rules. It SHALL stop when a round yields
fewer than 2 new rules OR shrinks the unmatched pool by less than 10%. It
SHALL hard-cap at 3 rounds regardless of shrink rate.

#### Scenario: Stops on fewer than 2 new rules

- **WHEN** a retest round yields only 1 new confirmed rule
- **THEN** the retest loop stops after that round

#### Scenario: Stops on less than 10% shrink

- **WHEN** a retest round shrinks the unmatched pool by less than 10%
- **THEN** the retest loop stops after that round, even if 2 or more rules were confirmed

#### Scenario: Hard cap of 3 rounds regardless of shrink

- **WHEN** three retest rounds have run and each still meets the continuation criteria (≥2 new rules and ≥10% shrink)
- **THEN** the retest loop stops after the third round regardless

### Requirement: Seed Intake at Judge Intake

The clutter-inventory seed candidates SHALL enter the calibration protocol at
judge intake. Full seed intake SHALL apply to every calibration run after
calibration pass #1; pass #1 uses the one-off seed hold-out described in the
`Calibration Pass Validation Criteria` requirement below.

#### Scenario: Seed candidates feed the judge step

- **WHEN** a calibration run (other than pass #1) begins
- **THEN** the clutter-inventory seed candidates are included as judge-intake evidence

### Requirement: Rule-Quality Test — Class Never Path

A proposed rule's glob SHALL name a recurring class of artifact, never an
identifier unique to a single instance. A glob that hardcodes a name
recurring by convention (e.g. `PRD.md`, `HANDOFF-*.md`,
`migration-report.md`) is acceptable. A glob that hardcodes a run-id, hash,
or bare timestamp unique to one instance is not acceptable. A rule that
currently matches only one file is acceptable; a rule that can, by
construction, only ever match one file is a failed generalization and SHALL
be flagged loudly rather than silently persisted.

#### Scenario: A convention-recurring name is acceptable

- **WHEN** a proposed rule's glob is `HANDOFF-*.md`
- **THEN** it passes the class-never-path test, since `HANDOFF-*` is a recurring naming convention, not a single instance

#### Scenario: An instance-unique identifier fails the test

- **WHEN** a proposed rule's glob hardcodes a specific run-id or hash string that can only ever identify one artifact
- **THEN** the rule fails the class-never-path test and is flagged loudly, not silently persisted

#### Scenario: One current match is fine; one-EVER match is not

- **WHEN** a proposed rule currently matches exactly one file
- **THEN** it is acceptable if the glob's structure could match future similarly-named files; it is flagged as a failed generalization if the glob's structure can never match any file but the one it names today

### Requirement: Rule-Quality Tie-Breaker — Prefer the Narrower Glob

When choosing between candidate globs of differing breadth for the same
cluster, `:calibrate` SHALL prefer the narrower glob. Too-narrow failure is
self-healing (clutter is merely left for a later round to catch); too-broad
failure is dangerous (a keeper file may be deleted and is not
self-healing).

#### Scenario: Narrower glob is chosen when both would satisfy the cluster

- **WHEN** two candidate globs both cover the sampled cluster, one narrower and one broader
- **THEN** the narrower glob is chosen

#### Scenario: Rationale is evidence quality, not readability alone

- **WHEN** justifying the narrower-glob preference
- **THEN** the reasoning cited is that too-narrow fails safe (self-healing) while too-broad fails dangerous (not self-healing), not merely stylistic preference

### Requirement: Calibration Pass Validation Criteria

A calibration pass SHALL be judged against: a precision hard gate (the pass
FAILS if any persisted rule's glob matches a protected path, regardless of
behavior tier — exploration-time `consult` verdicts on protected paths are
free and do not fail the pass); a recall floor of 8 of the 10 rows of the
project's clutter inventory, with 4 specific rows mandatory (missing any
mandatory row fails the pass); a one-off seed hold-out for calibration pass
#1 only (the sealed answer key is withheld from judge intake for pass #1;
every later run uses full seed intake); treatment of IGNORE-surface paths as
void, not a miss, against the recall floor; and the requirement that a
do-nothing pass cannot pass (the recall floor makes the pass falsifiable in
the finding direction).

#### Scenario: A rule matching a protected path fails the pass

- **WHEN** any persisted rule's glob matches a path in the fixed protected set
- **THEN** the calibration pass fails, regardless of whether that rule's tier was auto or confirm

#### Scenario: Consult verdicts on protected paths do not fail the pass

- **WHEN** the judge issues a `consult` verdict during exploration for a candidate touching a protected path, and that candidate is not persisted
- **THEN** the pass is not failed by that consult verdict

#### Scenario: Recall floor requires 8 of 10 with 4 mandatory

- **WHEN** a calibration pass is graded against the clutter inventory
- **THEN** it must recall at least 8 of the 10 inventory rows, and all 4 mandatory rows must be among them, or the pass fails

#### Scenario: Pass #1 seed hold-out is a one-off

- **WHEN** calibration pass #1 runs
- **THEN** the sealed answer key (the project's clutter-inventory rows) is withheld from judge intake; every subsequent calibration run instead uses full seed intake

#### Scenario: IGNORE-surface rows are void, not a miss

- **WHEN** grading recall against the clutter inventory and a row corresponds to an IGNORE-surface path
- **THEN** that row is excluded from the recall calculation entirely (void), not counted as a miss

#### Scenario: A do-nothing pass cannot pass

- **WHEN** a calibration pass persists zero rules
- **THEN** it fails the recall floor and therefore cannot pass