12 KiB

Raw Blame History

Spec: calibrate

ADDED Requirements

Requirement: Calibrate Clusters and Samples Over Unmatched Files

The :calibrate skill SHALL run over the unmatched-files pool (unmatched = unmanaged, per lifecycle-rulebook) as its candidate pool. It SHALL cluster unmatched paths by shape before nominating any rule, so that a proposed rule is authored against a cluster of similar paths rather than a single file.

Scenario: Calibrate operates only on the unmatched pool

WHEN :calibrate runs
THEN its candidate pool is exactly the set of files the rulebook currently leaves unmatched

Scenario: Rules are proposed against clusters, not single files

WHEN :calibrate nominates a candidate rule
THEN the nomination is derived from a cluster of similar unmatched paths, not from one instance in isolation

Requirement: Cheap-Model Nomination Produces Patterns, Never Exact-Instance Globs

For each sampled cluster, :calibrate SHALL dispatch a haiku subagent constrained to nominate a bare glob pattern plus a candidate lifetime. The haiku nomination SHALL be constrained to produce generalizable patterns; it SHALL NOT be accepted as final if it hardcodes an identifier unique to a single instance (a run-id, hash, or bare timestamp).

Scenario: Haiku nominates a glob and lifetime per cluster

WHEN a cluster of unmatched paths is sampled
THEN the haiku subagent returns a bare glob pattern and a candidate lifetime for that cluster

Scenario: Instance-unique nominations are not accepted as final

WHEN a haiku nomination's glob hardcodes a run-id, hash, or bare timestamp unique to one file
THEN it is not persisted as-is; it must be caught and generalized before the strong-model judgment step, per the rule-quality tests below

Requirement: Strong-Model Batched Judgment with Confirm/Reject/Amend/Consult

:calibrate SHALL dispatch one batched strong-model (Opus or Fable) judge subagent that gathers its own evidence (re-reading matched paths and checking near-miss boundaries) and authors final rule entries. The judge SHALL return one of four verdicts per nominated rule: confirm, reject, amend, or consult. consult SHALL be mandatory whenever an artifact's purpose is unclear (i.e., the judge cannot determine whether the artifact is regenerable or must be retained).

Scenario: Judge gathers its own evidence rather than trusting the nomination

WHEN the strong-model judge evaluates a haiku nomination
THEN it independently re-reads matched paths and checks near-miss boundaries rather than accepting the nomination's claims at face value

Scenario: Four verdicts are the only possible outcomes

WHEN the judge evaluates a nominated rule
THEN its verdict is exactly one of confirm, reject, amend, or consult

Scenario: Consult is mandatory when purpose is unclear

WHEN the judge cannot determine whether a clustered artifact type is regenerable or must be retained
THEN the verdict is consult, never confirm or reject

Requirement: Rule Report to the Human Before Persistence

Before any proposed rule is persisted, :calibrate SHALL present a rule report to the human containing, per proposed rule: the glob verbatim exactly as it would be persisted; every path it currently matches (or a capped sample plus a total count); the near-miss boundary — paths that do NOT match despite looking similar; the lifetime and behavior tier (auto vs confirm); and a plain-language explanation of what the artifact is and why it is clutter. No rule SHALL be persisted before this report has been shown.

Scenario: The report shows the exact persisted glob

WHEN a rule report is generated for a proposed rule
THEN the glob shown is character-for-character identical to what would be written to the rulebook file

Scenario: The report shows current matches with a capped sample

WHEN a proposed rule matches more paths than the display cap
THEN the report shows a capped sample of matched paths plus the total count of all matches

Scenario: The report shows the near-miss boundary

WHEN a proposed rule's glob narrowly excludes similar-looking paths
THEN the report explicitly lists those near-miss non-matching paths, so a boundary bug (e.g. a glob silently missing a sibling path) is visible before persistence

Scenario: No rule is persisted before the report is shown

WHEN :calibrate has generated proposed rules
THEN it does not write any rule to a rulebook file until the human has seen the rule report for it

Requirement: Persistence Rules by Scope

Project-rulebook writes SHALL land on judge confirmation once the human has reviewed the rule report. Global-rulebook writes (writing into plugins/os-doc-hygiene/rulebook.json) SHALL additionally require explicit human gating, distinct from project-rule confirmation, since it is a cross-repo write into cc-os. Rule removals SHALL be HITL-only in all cases, with recorded reasoning, regardless of scope.

Scenario: Project rule persists on judge confirmation plus report review

WHEN the judge verdict is confirm for a project-scoped rule and the human has reviewed its rule report
THEN the rule is written to the project's .dochygiene-rules.json

Scenario: Global rulebook writes require an additional explicit gate

WHEN a proposed rule would be written to the global rulebook.json
THEN a distinct human confirmation for the cross-repo write is required, beyond the project-rule confirmation step

Scenario: Rule removal is always HITL-only

WHEN any rule (project or global) is proposed for removal
THEN the removal happens only via explicit human instruction, with the reasoning recorded, never as an automatic side effect of a calibration pass

Requirement: Retest Loop with Stop Conditions and Hard Cap

:calibrate SHALL re-run its clustering pass against the shrunk unmatched pool after each round of persisted rules. It SHALL stop when a round yields fewer than 2 new rules OR shrinks the unmatched pool by less than 10%. It SHALL hard-cap at 3 rounds regardless of shrink rate.

Scenario: Stops on fewer than 2 new rules

WHEN a retest round yields only 1 new confirmed rule
THEN the retest loop stops after that round

Scenario: Stops on less than 10% shrink

WHEN a retest round shrinks the unmatched pool by less than 10%
THEN the retest loop stops after that round, even if 2 or more rules were confirmed

Scenario: Hard cap of 3 rounds regardless of shrink

WHEN three retest rounds have run and each still meets the continuation criteria (≥2 new rules and ≥10% shrink)
THEN the retest loop stops after the third round regardless

Requirement: Seed Intake at Judge Intake

The clutter-inventory seed candidates SHALL enter the calibration protocol at judge intake. Full seed intake SHALL apply to every calibration run after calibration pass #1; pass #1 uses the one-off seed hold-out described in the Calibration Pass Validation Criteria requirement below.

Scenario: Seed candidates feed the judge step

WHEN a calibration run (other than pass #1) begins
THEN the clutter-inventory seed candidates are included as judge-intake evidence

Requirement: Rule-Quality Test — Class Never Path

A proposed rule's glob SHALL name a recurring class of artifact, never an identifier unique to a single instance. A glob that hardcodes a name recurring by convention (e.g. PRD.md, HANDOFF-*.md, migration-report.md) is acceptable. A glob that hardcodes a run-id, hash, or bare timestamp unique to one instance is not acceptable. A rule that currently matches only one file is acceptable; a rule that can, by construction, only ever match one file is a failed generalization and SHALL be flagged loudly rather than silently persisted.

Scenario: A convention-recurring name is acceptable

WHEN a proposed rule's glob is HANDOFF-*.md
THEN it passes the class-never-path test, since HANDOFF-* is a recurring naming convention, not a single instance

Scenario: An instance-unique identifier fails the test

WHEN a proposed rule's glob hardcodes a specific run-id or hash string that can only ever identify one artifact
THEN the rule fails the class-never-path test and is flagged loudly, not silently persisted

Scenario: One current match is fine; one-EVER match is not

WHEN a proposed rule currently matches exactly one file
THEN it is acceptable if the glob's structure could match future similarly-named files; it is flagged as a failed generalization if the glob's structure can never match any file but the one it names today

Requirement: Rule-Quality Tie-Breaker — Prefer the Narrower Glob

When choosing between candidate globs of differing breadth for the same cluster, :calibrate SHALL prefer the narrower glob. Too-narrow failure is self-healing (clutter is merely left for a later round to catch); too-broad failure is dangerous (a keeper file may be deleted and is not self-healing).

Scenario: Narrower glob is chosen when both would satisfy the cluster

WHEN two candidate globs both cover the sampled cluster, one narrower and one broader
THEN the narrower glob is chosen

Scenario: Rationale is evidence quality, not readability alone

WHEN justifying the narrower-glob preference
THEN the reasoning cited is that too-narrow fails safe (self-healing) while too-broad fails dangerous (not self-healing), not merely stylistic preference

Requirement: Calibration Pass Validation Criteria

A calibration pass SHALL be judged against: a precision hard gate (the pass FAILS if any persisted rule's glob matches a protected path, regardless of behavior tier — exploration-time consult verdicts on protected paths are free and do not fail the pass); a recall floor of 8 of the 10 rows of the project's clutter inventory, with 4 specific rows mandatory (missing any mandatory row fails the pass); a one-off seed hold-out for calibration pass #1 only (the sealed answer key is withheld from judge intake for pass #1; every later run uses full seed intake); treatment of IGNORE-surface paths as void, not a miss, against the recall floor; and the requirement that a do-nothing pass cannot pass (the recall floor makes the pass falsifiable in the finding direction).

Scenario: A rule matching a protected path fails the pass

WHEN any persisted rule's glob matches a path in the fixed protected set
THEN the calibration pass fails, regardless of whether that rule's tier was auto or confirm

Scenario: Consult verdicts on protected paths do not fail the pass

WHEN the judge issues a consult verdict during exploration for a candidate touching a protected path, and that candidate is not persisted
THEN the pass is not failed by that consult verdict

Scenario: Recall floor requires 8 of 10 with 4 mandatory

WHEN a calibration pass is graded against the clutter inventory
THEN it must recall at least 8 of the 10 inventory rows, and all 4 mandatory rows must be among them, or the pass fails

Scenario: Pass #1 seed hold-out is a one-off

WHEN calibration pass #1 runs
THEN the sealed answer key (the project's clutter-inventory rows) is withheld from judge intake; every subsequent calibration run instead uses full seed intake

Scenario: IGNORE-surface rows are void, not a miss

WHEN grading recall against the clutter inventory and a row corresponds to an IGNORE-surface path
THEN that row is excluded from the recall calculation entirely (void), not counted as a miss

Scenario: A do-nothing pass cannot pass

WHEN a calibration pass persists zero rules
THEN it fails the recall floor and therefore cannot pass

12 KiB Raw Blame History