18 KiB

Raw Blame History

Spec: calibrate

Purpose

Defines the :calibrate skill, which proposes new lifecycle rulebook rules by clustering the pool of files the rulebook currently leaves unmatched, having a cheap model nominate generalizable glob patterns, and having a strong model judge and report those nominations to a human before any persistence.

Requirements

Requirement: Calibrate Clusters and Samples Over Unmatched Files

The :calibrate skill SHALL run over the unmatched-files pool (unmatched = unmanaged, per lifecycle-rulebook) as its candidate pool. It SHALL cluster unmatched paths by shape before nominating any rule, so that a proposed rule is authored against a cluster of similar paths rather than a single file.

Scenario: Calibrate operates only on the unmatched pool

WHEN :calibrate runs
THEN its candidate pool is exactly the set of files the rulebook currently leaves unmatched

Scenario: Rules are proposed against clusters, not single files

WHEN :calibrate nominates a candidate rule
THEN the nomination is derived from a cluster of similar unmatched paths, not from one instance in isolation

Requirement: Cheap-Model Nomination Produces Patterns, Never Exact-Instance Globs

For each sampled cluster, :calibrate SHALL dispatch a haiku subagent constrained to nominate a bare glob pattern plus a candidate lifetime. The haiku nomination SHALL be constrained to produce generalizable patterns; it SHALL NOT be accepted as final if it hardcodes an identifier unique to a single instance (a run-id, hash, or bare timestamp).

Scenario: Haiku nominates a glob and lifetime per cluster

WHEN a cluster of unmatched paths is sampled
THEN the haiku subagent returns a bare glob pattern and a candidate lifetime for that cluster

Scenario: Instance-unique nominations are not accepted as final

WHEN a haiku nomination's glob hardcodes a run-id, hash, or bare timestamp unique to one file
THEN it is not persisted as-is; it must be caught and generalized before the strong-model judgment step, per the rule-quality tests below

Requirement: Strong-Model Batched Judgment with Confirm/Reject/Amend/Consult

:calibrate SHALL dispatch one batched strong-model (Opus or Fable) judge subagent that gathers its own evidence (re-reading matched paths and checking near-miss boundaries) and authors final rule entries. The judge SHALL return one of four verdicts per nominated rule: confirm, reject, amend, or consult. consult SHALL be mandatory whenever an artifact's purpose is unclear (i.e., the judge cannot determine whether the artifact is regenerable or must be retained).

Scenario: Judge gathers its own evidence rather than trusting the nomination

WHEN the strong-model judge evaluates a haiku nomination
THEN it independently re-reads matched paths and checks near-miss boundaries rather than accepting the nomination's claims at face value

Scenario: Four verdicts are the only possible outcomes

WHEN the judge evaluates a nominated rule
THEN its verdict is exactly one of confirm, reject, amend, or consult

Scenario: Consult is mandatory when purpose is unclear

WHEN the judge cannot determine whether a clustered artifact type is regenerable or must be retained
THEN the verdict is consult, never confirm or reject

Requirement: Rule Report to the Human Before Persistence

Before any proposed rule is persisted, :calibrate SHALL present a rule report to the human containing, per proposed rule: the glob verbatim exactly as it would be persisted; every path it currently matches (or a capped sample plus a total count); the near-miss boundary — paths that do NOT match despite looking similar; the lifetime and behavior tier (auto vs confirm); and a plain-language explanation of what the artifact is and why it is clutter. No rule SHALL be persisted before this report has been shown.

Scenario: The report shows the exact persisted glob

WHEN a rule report is generated for a proposed rule
THEN the glob shown is character-for-character identical to what would be written to the rulebook file

Scenario: The report shows current matches with a capped sample

WHEN a proposed rule matches more paths than the display cap
THEN the report shows a capped sample of matched paths plus the total count of all matches

Scenario: The report shows the near-miss boundary

WHEN a proposed rule's glob narrowly excludes similar-looking paths
THEN the report explicitly lists those near-miss non-matching paths, so a boundary bug (e.g. a glob silently missing a sibling path) is visible before persistence

Scenario: No rule is persisted before the report is shown

WHEN :calibrate has generated proposed rules
THEN it does not write any rule to a rulebook file until the human has seen the rule report for it

Requirement: Persistence Rules by Scope

Project-rulebook writes SHALL land on judge confirmation once the human has reviewed the rule report. Global-rulebook writes (writing into plugins/os-doc-hygiene/rulebook.json) SHALL additionally require explicit human gating, distinct from project-rule confirmation, since it is a cross-repo write into cc-os. Rule removals SHALL be HITL-only in all cases, with recorded reasoning, regardless of scope. Every settled verdict SHALL persist: judge keep verdicts become ordinary lifetime: keep rules in rules (including exact-path singletons, per the keep-tier relaxation); human declines at the rule report persist as rejected entries with rejected_by: "human"; open consult verdicts persist to nominations.consults, deduped by glob at write time. All persistence SHALL go through the canonical rules-file writer.

Scenario: Project rule persists on judge confirmation plus report review

WHEN the judge verdict is confirm for a project-scoped rule and the human has reviewed its rule report
THEN the rule is written to the project's .dochygiene-rules.json

Scenario: Global rulebook writes require an additional explicit gate

WHEN a proposed rule would be written to the global rulebook.json
THEN a distinct human confirmation for the cross-repo write is required, beyond the project-rule confirmation step

Scenario: Rule removal is always HITL-only

WHEN any rule (project or global) is proposed for removal
THEN the removal happens only via explicit human instruction, with the reasoning recorded, never as an automatic side effect of a calibration pass

Scenario: A keep verdict persists as a plain keep rule

WHEN the judge's settled verdict for a cluster is that the artifacts must be retained
THEN an ordinary lifetime: keep rule is persisted to rules — removing the cluster from future calibrate pools and protecting future matches — even when the glob is an exact-path singleton

Scenario: A human decline persists as a rejection

WHEN the human declines a judge-confirmed rule at the rule report
THEN a rejected entry with rejected_by: "human" is written, so a later haiku round cannot re-nominate the identical glob+lifetime without the judge knowing

Requirement: Retest Loop with Stop Conditions and Hard Cap

:calibrate SHALL re-run its clustering pass against the shrunk unmatched pool after each round of persisted rules. It SHALL stop when a round yields fewer than 2 new rules OR shrinks the unmatched pool by less than 10%. It SHALL hard-cap at 3 rounds regardless of shrink rate.

Scenario: Stops on fewer than 2 new rules

WHEN a retest round yields only 1 new confirmed rule
THEN the retest loop stops after that round

Scenario: Stops on less than 10% shrink

WHEN a retest round shrinks the unmatched pool by less than 10%
THEN the retest loop stops after that round, even if 2 or more rules were confirmed

Scenario: Hard cap of 3 rounds regardless of shrink

WHEN three retest rounds have run and each still meets the continuation criteria (≥2 new rules and ≥10% shrink)
THEN the retest loop stops after the third round regardless

Requirement: Seed Intake at Judge Intake

The clutter-inventory seed candidates SHALL enter the calibration protocol at judge intake. Full seed intake SHALL apply to every calibration run after calibration pass #1; pass #1 uses the one-off seed hold-out described in the Calibration Pass Validation Criteria requirement below.

Scenario: Seed candidates feed the judge step

WHEN a calibration run (other than pass #1) begins
THEN the clutter-inventory seed candidates are included as judge-intake evidence

Requirement: Rule-Quality Test — Class Never Path

A proposed rule's glob SHALL name a recurring class of artifact, never an identifier unique to a single instance. A glob that hardcodes a name recurring by convention (e.g. PRD.md, HANDOFF-*.md, migration-report.md) is acceptable. A glob that hardcodes a run-id, hash, or bare timestamp unique to one instance is not acceptable. A rule that currently matches only one file is acceptable; a rule that can, by construction, only ever match one file is a failed generalization and SHALL be flagged loudly rather than silently persisted. Keep-tier relaxation: exact-path/instance globs ARE acceptable for lifetime: keep entries only — this test exists to prevent bad deletion rules, and a singleton keep merely protects; instance globs remain forbidden for temporary and delete-once-served. When the near-miss boundary check reveals sibling artifacts a glob misses, the fix SHALL be to enumerate the conventional prefixes as separate rule entries, never to widen the glob to the containing directory; container-claiming globs are justified ONLY when the directory is wholly machine-owned (e.g. plugins/*/.pytest_cache/).

Scenario: A convention-recurring name is acceptable

WHEN a proposed rule's glob is HANDOFF-*.md
THEN it passes the class-never-path test, since HANDOFF-* is a recurring naming convention, not a single instance

Scenario: An instance-unique identifier fails the test

WHEN a proposed rule's glob hardcodes a specific run-id or hash string that can only ever identify one artifact
THEN the rule fails the class-never-path test and is flagged loudly, not silently persisted

Scenario: One current match is fine; one-EVER match is not

WHEN a proposed rule currently matches exactly one file
THEN it is acceptable if the glob's structure could match future similarly-named files; it is flagged as a failed generalization if the glob's structure can never match any file but the one it names today

Scenario: A singleton keep passes under the keep-tier relaxation

WHEN a proposed rule is docs/research/clutter-pattern-inventory.md -> keep
THEN it passes despite being an exact-path instance glob, because the keep tier only protects; the same glob with lifetime temporary or delete-once-served fails

Scenario: Missed siblings are enumerated, never widened to the container

WHEN the boundary check shows autoresearch/classic-*/ misses sibling runs under autoresearch/improve-*/
THEN the fix is a second rule entry autoresearch/improve-*/ (same cluster), never a widening to autoresearch/*/ or autoresearch/**, so keep-worthy content can still live in the container without a counter-rule

Requirement: Rule-Quality Tie-Breaker — Prefer the Narrower Glob

When choosing between candidate globs of differing breadth for the same cluster, :calibrate SHALL prefer the narrower glob. Too-narrow failure is self-healing (clutter is merely left for a later round to catch); too-broad failure is dangerous (a keeper file may be deleted and is not self-healing).

Scenario: Narrower glob is chosen when both would satisfy the cluster

WHEN two candidate globs both cover the sampled cluster, one narrower and one broader
THEN the narrower glob is chosen

Scenario: Rationale is evidence quality, not readability alone

WHEN justifying the narrower-glob preference
THEN the reasoning cited is that too-narrow fails safe (self-healing) while too-broad fails dangerous (not self-healing), not merely stylistic preference

Requirement: Calibration Pass Validation Criteria

A calibration pass SHALL be judged against: a precision hard gate (the pass FAILS if any persisted rule's glob matches a protected path, regardless of behavior tier — exploration-time consult verdicts on protected paths are free and do not fail the pass); a recall floor of 8 of the 10 rows of the project's clutter inventory, with 4 specific rows mandatory (missing any mandatory row fails the pass); a one-off seed hold-out for calibration pass #1 only (the sealed answer key is withheld from judge intake for pass #1; every later run uses full seed intake); treatment of IGNORE-surface paths as void, not a miss, against the recall floor; and the requirement that a do-nothing pass cannot pass (the recall floor makes the pass falsifiable in the finding direction).

Scenario: A rule matching a protected path fails the pass

WHEN any persisted rule's glob matches a path in the fixed protected set
THEN the calibration pass fails, regardless of whether that rule's tier was auto or confirm

Scenario: Consult verdicts on protected paths do not fail the pass

WHEN the judge issues a consult verdict during exploration for a candidate touching a protected path, and that candidate is not persisted
THEN the pass is not failed by that consult verdict

Scenario: Recall floor requires 8 of 10 with 4 mandatory

WHEN a calibration pass is graded against the clutter inventory
THEN it must recall at least 8 of the 10 inventory rows, and all 4 mandatory rows must be among them, or the pass fails

Scenario: Pass #1 seed hold-out is a one-off

WHEN calibration pass #1 runs
THEN the sealed answer key (the project's clutter-inventory rows) is withheld from judge intake; every subsequent calibration run instead uses full seed intake

Scenario: IGNORE-surface rows are void, not a miss

WHEN grading recall against the clutter inventory and a row corresponds to an IGNORE-surface path
THEN that row is excluded from the recall calculation entirely (void), not counted as a miss

Scenario: A do-nothing pass cannot pass

WHEN a calibration pass persists zero rules
THEN it fails the recall floor and therefore cannot pass

Requirement: Deterministic Nomination Intake Filter

Between cheap-model nomination and strong-model judgment, :calibrate SHALL run a deterministic NominationIntakeFilter (in calibrate_helpers.py, no model — invariant #6). A nomination whose glob+lifetime exactly equals a rejected entry SHALL be dropped before the judge and logged in the run summary. Surviving nominations SHALL be annotated with every related rejection, where related means the two globs' match sets intersect on the current shortlist (deterministic, computed from the scan). The annotations plus all open consults SHALL enter the judge prompt as its "Nominations memory" input section; related rejections are context for the judge, never a veto.

Scenario: Exact glob+lifetime repeat is dropped before the judge

WHEN haiku nominates docs/research/** -> temporary and nominations.rejected contains an entry with glob docs/research/** and lifetime temporary
THEN the nomination is dropped at intake, never reaches the judge, and the drop is logged in the run summary

Scenario: A variant flows through annotated, not blocked

WHEN haiku nominates docs/research/drafts/** -> temporary and nominations.rejected contains docs/research/** -> temporary, and the two globs' match sets intersect on the current shortlist
THEN the nomination proceeds to the judge carrying the related rejection (its why and consider_instead) as context, and the judge may still confirm it

Scenario: Open consults always reach the judge prompt

WHEN nominations.consults is non-empty at intake time
THEN every open consult is included in the judge prompt's "Nominations memory" section, regardless of what haiku nominated this round

Requirement: Consult Persistence and Resurfacing

Open consult verdicts SHALL persist to nominations.consults (deduped by glob at write time) rather than dying with the run. Consults SHALL resurface in :calibrate only — :check and :clean are unchanged — appearing in the judge prompt and as an "Open consults" section of the rule report. A consult SHALL exit in exactly one of three ways: (a) a human answer settles the purpose — a normal rule is persisted and the consult entry deleted; (b) the human deems it not rule-worthy — the entry is rewritten into rejected with rejected_by: "human" and the human's why; (c) the human defers — the entry stays and resurfaces next run. New rejections and consults SHALL appear in the rule report but are not individually gated — they are memory, not deletion authority.

Scenario: A consult survives the run and resurfaces

WHEN a judge verdict is consult and the run ends without a human answer
THEN the consult is written to nominations.consults, and the next :calibrate run surfaces it in both the judge prompt and the report's "Open consults" section

Scenario: An answered consult becomes a rule and disappears

WHEN the human answers an open consult in a way that settles the artifact's purpose
THEN a normal rule is persisted through the standard report flow and the consult entry is deleted in the same write

Scenario: A declined consult becomes a human rejection

WHEN the human answers that an open consult's artifact class is not rule-worthy
THEN the consult entry is rewritten into nominations.rejected with rejected_by: "human" and the stated reason

Scenario: Consults never surface outside calibrate

WHEN :check or :clean runs against a project with open consults
THEN their behavior is unchanged — consults neither appear in output nor affect any classification

18 KiB Raw Blame History