343 lines
18 KiB
Markdown
343 lines
18 KiB
Markdown
# Spec: calibrate
|
|
|
|
## Purpose
|
|
|
|
Defines the `:calibrate` skill, which proposes new lifecycle rulebook rules by
|
|
clustering the pool of files the rulebook currently leaves unmatched, having a
|
|
cheap model nominate generalizable glob patterns, and having a strong model
|
|
judge and report those nominations to a human before any persistence.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: Calibrate Clusters and Samples Over Unmatched Files
|
|
|
|
The `:calibrate` skill SHALL run over the unmatched-files pool (unmatched =
|
|
unmanaged, per `lifecycle-rulebook`) as its candidate pool. It SHALL cluster
|
|
unmatched paths by shape before nominating any rule, so that a proposed rule
|
|
is authored against a cluster of similar paths rather than a single file.
|
|
|
|
#### Scenario: Calibrate operates only on the unmatched pool
|
|
|
|
- **WHEN** `:calibrate` runs
|
|
- **THEN** its candidate pool is exactly the set of files the rulebook currently leaves unmatched
|
|
|
|
#### Scenario: Rules are proposed against clusters, not single files
|
|
|
|
- **WHEN** `:calibrate` nominates a candidate rule
|
|
- **THEN** the nomination is derived from a cluster of similar unmatched paths, not from one instance in isolation
|
|
|
|
### Requirement: Cheap-Model Nomination Produces Patterns, Never Exact-Instance Globs
|
|
|
|
For each sampled cluster, `:calibrate` SHALL dispatch a haiku subagent
|
|
constrained to nominate a bare glob pattern plus a candidate lifetime. The
|
|
haiku nomination SHALL be constrained to produce generalizable patterns; it
|
|
SHALL NOT be accepted as final if it hardcodes an identifier unique to a
|
|
single instance (a run-id, hash, or bare timestamp).
|
|
|
|
#### Scenario: Haiku nominates a glob and lifetime per cluster
|
|
|
|
- **WHEN** a cluster of unmatched paths is sampled
|
|
- **THEN** the haiku subagent returns a bare glob pattern and a candidate lifetime for that cluster
|
|
|
|
#### Scenario: Instance-unique nominations are not accepted as final
|
|
|
|
- **WHEN** a haiku nomination's glob hardcodes a run-id, hash, or bare timestamp unique to one file
|
|
- **THEN** it is not persisted as-is; it must be caught and generalized before the strong-model judgment step, per the rule-quality tests below
|
|
|
|
### Requirement: Strong-Model Batched Judgment with Confirm/Reject/Amend/Consult
|
|
|
|
`:calibrate` SHALL dispatch one batched strong-model (Opus or Fable) judge
|
|
subagent that gathers its own evidence (re-reading matched paths and
|
|
checking near-miss boundaries) and authors final rule entries. The judge
|
|
SHALL return one of four verdicts per nominated rule: `confirm`, `reject`,
|
|
`amend`, or `consult`. `consult` SHALL be mandatory whenever an artifact's
|
|
purpose is unclear (i.e., the judge cannot determine whether the artifact is
|
|
regenerable or must be retained).
|
|
|
|
#### Scenario: Judge gathers its own evidence rather than trusting the nomination
|
|
|
|
- **WHEN** the strong-model judge evaluates a haiku nomination
|
|
- **THEN** it independently re-reads matched paths and checks near-miss boundaries rather than accepting the nomination's claims at face value
|
|
|
|
#### Scenario: Four verdicts are the only possible outcomes
|
|
|
|
- **WHEN** the judge evaluates a nominated rule
|
|
- **THEN** its verdict is exactly one of `confirm`, `reject`, `amend`, or `consult`
|
|
|
|
#### Scenario: Consult is mandatory when purpose is unclear
|
|
|
|
- **WHEN** the judge cannot determine whether a clustered artifact type is regenerable or must be retained
|
|
- **THEN** the verdict is `consult`, never `confirm` or `reject`
|
|
|
|
### Requirement: Rule Report to the Human Before Persistence
|
|
|
|
Before any proposed rule is persisted, `:calibrate` SHALL present a rule
|
|
report to the human containing, per proposed rule: the glob verbatim exactly
|
|
as it would be persisted; every path it currently matches (or a capped
|
|
sample plus a total count); the near-miss boundary — paths that do NOT match
|
|
despite looking similar; the lifetime and behavior tier (auto vs confirm);
|
|
and a plain-language explanation of what the artifact is and why it is
|
|
clutter. No rule SHALL be persisted before this report has been shown.
|
|
|
|
#### Scenario: The report shows the exact persisted glob
|
|
|
|
- **WHEN** a rule report is generated for a proposed rule
|
|
- **THEN** the glob shown is character-for-character identical to what would be written to the rulebook file
|
|
|
|
#### Scenario: The report shows current matches with a capped sample
|
|
|
|
- **WHEN** a proposed rule matches more paths than the display cap
|
|
- **THEN** the report shows a capped sample of matched paths plus the total count of all matches
|
|
|
|
#### Scenario: The report shows the near-miss boundary
|
|
|
|
- **WHEN** a proposed rule's glob narrowly excludes similar-looking paths
|
|
- **THEN** the report explicitly lists those near-miss non-matching paths, so a boundary bug (e.g. a glob silently missing a sibling path) is visible before persistence
|
|
|
|
#### Scenario: No rule is persisted before the report is shown
|
|
|
|
- **WHEN** `:calibrate` has generated proposed rules
|
|
- **THEN** it does not write any rule to a rulebook file until the human has seen the rule report for it
|
|
|
|
### Requirement: Persistence Rules by Scope
|
|
|
|
Project-rulebook writes SHALL land on judge confirmation once the human has
|
|
reviewed the rule report. Global-rulebook writes (writing into
|
|
`plugins/os-doc-hygiene/rulebook.json`) SHALL additionally require explicit
|
|
human gating, distinct from project-rule confirmation, since it is a
|
|
cross-repo write into cc-os. Rule removals SHALL be HITL-only in all cases,
|
|
with recorded reasoning, regardless of scope. Every settled verdict SHALL
|
|
persist: judge `keep` verdicts become ordinary `lifetime: keep` rules in
|
|
`rules` (including exact-path singletons, per the keep-tier relaxation);
|
|
human declines at the rule report persist as `rejected` entries with
|
|
`rejected_by: "human"`; open `consult` verdicts persist to
|
|
`nominations.consults`, deduped by glob at write time. All persistence SHALL
|
|
go through the canonical rules-file writer.
|
|
|
|
#### Scenario: Project rule persists on judge confirmation plus report review
|
|
|
|
- **WHEN** the judge verdict is `confirm` for a project-scoped rule and the human has reviewed its rule report
|
|
- **THEN** the rule is written to the project's `.dochygiene-rules.json`
|
|
|
|
#### Scenario: Global rulebook writes require an additional explicit gate
|
|
|
|
- **WHEN** a proposed rule would be written to the global `rulebook.json`
|
|
- **THEN** a distinct human confirmation for the cross-repo write is required, beyond the project-rule confirmation step
|
|
|
|
#### Scenario: Rule removal is always HITL-only
|
|
|
|
- **WHEN** any rule (project or global) is proposed for removal
|
|
- **THEN** the removal happens only via explicit human instruction, with the reasoning recorded, never as an automatic side effect of a calibration pass
|
|
|
|
#### Scenario: A keep verdict persists as a plain keep rule
|
|
|
|
- **WHEN** the judge's settled verdict for a cluster is that the artifacts must be retained
|
|
- **THEN** an ordinary `lifetime: keep` rule is persisted to `rules` — removing the cluster from future calibrate pools and protecting future matches — even when the glob is an exact-path singleton
|
|
|
|
#### Scenario: A human decline persists as a rejection
|
|
|
|
- **WHEN** the human declines a judge-confirmed rule at the rule report
|
|
- **THEN** a `rejected` entry with `rejected_by: "human"` is written, so a later haiku round cannot re-nominate the identical glob+lifetime without the judge knowing
|
|
|
|
### Requirement: Retest Loop with Stop Conditions and Hard Cap
|
|
|
|
`:calibrate` SHALL re-run its clustering pass against the shrunk unmatched
|
|
pool after each round of persisted rules. It SHALL stop when a round yields
|
|
fewer than 2 new rules OR shrinks the unmatched pool by less than 10%. It
|
|
SHALL hard-cap at 3 rounds regardless of shrink rate.
|
|
|
|
#### Scenario: Stops on fewer than 2 new rules
|
|
|
|
- **WHEN** a retest round yields only 1 new confirmed rule
|
|
- **THEN** the retest loop stops after that round
|
|
|
|
#### Scenario: Stops on less than 10% shrink
|
|
|
|
- **WHEN** a retest round shrinks the unmatched pool by less than 10%
|
|
- **THEN** the retest loop stops after that round, even if 2 or more rules were confirmed
|
|
|
|
#### Scenario: Hard cap of 3 rounds regardless of shrink
|
|
|
|
- **WHEN** three retest rounds have run and each still meets the continuation criteria (≥2 new rules and ≥10% shrink)
|
|
- **THEN** the retest loop stops after the third round regardless
|
|
|
|
### Requirement: Seed Intake at Judge Intake
|
|
|
|
The clutter-inventory seed candidates SHALL enter the calibration protocol at
|
|
judge intake. Full seed intake SHALL apply to every calibration run after
|
|
calibration pass #1; pass #1 uses the one-off seed hold-out described in the
|
|
`Calibration Pass Validation Criteria` requirement below.
|
|
|
|
#### Scenario: Seed candidates feed the judge step
|
|
|
|
- **WHEN** a calibration run (other than pass #1) begins
|
|
- **THEN** the clutter-inventory seed candidates are included as judge-intake evidence
|
|
|
|
### Requirement: Rule-Quality Test — Class Never Path
|
|
|
|
A proposed rule's glob SHALL name a recurring class of artifact, never an
|
|
identifier unique to a single instance. A glob that hardcodes a name
|
|
recurring by convention (e.g. `PRD.md`, `HANDOFF-*.md`,
|
|
`migration-report.md`) is acceptable. A glob that hardcodes a run-id, hash,
|
|
or bare timestamp unique to one instance is not acceptable. A rule that
|
|
currently matches only one file is acceptable; a rule that can, by
|
|
construction, only ever match one file is a failed generalization and SHALL
|
|
be flagged loudly rather than silently persisted. **Keep-tier relaxation:**
|
|
exact-path/instance globs ARE acceptable for `lifetime: keep` entries only —
|
|
this test exists to prevent bad deletion rules, and a singleton keep merely
|
|
protects; instance globs remain forbidden for `temporary` and
|
|
`delete-once-served`. When the near-miss boundary check reveals sibling
|
|
artifacts a glob misses, the fix SHALL be to enumerate the conventional
|
|
prefixes as separate rule entries, never to widen the glob to the containing
|
|
directory; container-claiming globs are justified ONLY when the directory is
|
|
wholly machine-owned (e.g. `plugins/*/.pytest_cache/`).
|
|
|
|
#### Scenario: A convention-recurring name is acceptable
|
|
|
|
- **WHEN** a proposed rule's glob is `HANDOFF-*.md`
|
|
- **THEN** it passes the class-never-path test, since `HANDOFF-*` is a recurring naming convention, not a single instance
|
|
|
|
#### Scenario: An instance-unique identifier fails the test
|
|
|
|
- **WHEN** a proposed rule's glob hardcodes a specific run-id or hash string that can only ever identify one artifact
|
|
- **THEN** the rule fails the class-never-path test and is flagged loudly, not silently persisted
|
|
|
|
#### Scenario: One current match is fine; one-EVER match is not
|
|
|
|
- **WHEN** a proposed rule currently matches exactly one file
|
|
- **THEN** it is acceptable if the glob's structure could match future similarly-named files; it is flagged as a failed generalization if the glob's structure can never match any file but the one it names today
|
|
|
|
#### Scenario: A singleton keep passes under the keep-tier relaxation
|
|
|
|
- **WHEN** a proposed rule is `docs/research/clutter-pattern-inventory.md -> keep`
|
|
- **THEN** it passes despite being an exact-path instance glob, because the keep tier only protects; the same glob with lifetime `temporary` or `delete-once-served` fails
|
|
|
|
#### Scenario: Missed siblings are enumerated, never widened to the container
|
|
|
|
- **WHEN** the boundary check shows `autoresearch/classic-*/` misses sibling runs under `autoresearch/improve-*/`
|
|
- **THEN** the fix is a second rule entry `autoresearch/improve-*/` (same cluster), never a widening to `autoresearch/*/` or `autoresearch/**`, so keep-worthy content can still live in the container without a counter-rule
|
|
|
|
### Requirement: Rule-Quality Tie-Breaker — Prefer the Narrower Glob
|
|
|
|
When choosing between candidate globs of differing breadth for the same
|
|
cluster, `:calibrate` SHALL prefer the narrower glob. Too-narrow failure is
|
|
self-healing (clutter is merely left for a later round to catch); too-broad
|
|
failure is dangerous (a keeper file may be deleted and is not
|
|
self-healing).
|
|
|
|
#### Scenario: Narrower glob is chosen when both would satisfy the cluster
|
|
|
|
- **WHEN** two candidate globs both cover the sampled cluster, one narrower and one broader
|
|
- **THEN** the narrower glob is chosen
|
|
|
|
#### Scenario: Rationale is evidence quality, not readability alone
|
|
|
|
- **WHEN** justifying the narrower-glob preference
|
|
- **THEN** the reasoning cited is that too-narrow fails safe (self-healing) while too-broad fails dangerous (not self-healing), not merely stylistic preference
|
|
|
|
### Requirement: Calibration Pass Validation Criteria
|
|
|
|
A calibration pass SHALL be judged against: a precision hard gate (the pass
|
|
FAILS if any persisted rule's glob matches a protected path, regardless of
|
|
behavior tier — exploration-time `consult` verdicts on protected paths are
|
|
free and do not fail the pass); a recall floor of 8 of the 10 rows of the
|
|
project's clutter inventory, with 4 specific rows mandatory (missing any
|
|
mandatory row fails the pass); a one-off seed hold-out for calibration pass
|
|
#1 only (the sealed answer key is withheld from judge intake for pass #1;
|
|
every later run uses full seed intake); treatment of IGNORE-surface paths as
|
|
void, not a miss, against the recall floor; and the requirement that a
|
|
do-nothing pass cannot pass (the recall floor makes the pass falsifiable in
|
|
the finding direction).
|
|
|
|
#### Scenario: A rule matching a protected path fails the pass
|
|
|
|
- **WHEN** any persisted rule's glob matches a path in the fixed protected set
|
|
- **THEN** the calibration pass fails, regardless of whether that rule's tier was auto or confirm
|
|
|
|
#### Scenario: Consult verdicts on protected paths do not fail the pass
|
|
|
|
- **WHEN** the judge issues a `consult` verdict during exploration for a candidate touching a protected path, and that candidate is not persisted
|
|
- **THEN** the pass is not failed by that consult verdict
|
|
|
|
#### Scenario: Recall floor requires 8 of 10 with 4 mandatory
|
|
|
|
- **WHEN** a calibration pass is graded against the clutter inventory
|
|
- **THEN** it must recall at least 8 of the 10 inventory rows, and all 4 mandatory rows must be among them, or the pass fails
|
|
|
|
#### Scenario: Pass #1 seed hold-out is a one-off
|
|
|
|
- **WHEN** calibration pass #1 runs
|
|
- **THEN** the sealed answer key (the project's clutter-inventory rows) is withheld from judge intake; every subsequent calibration run instead uses full seed intake
|
|
|
|
#### Scenario: IGNORE-surface rows are void, not a miss
|
|
|
|
- **WHEN** grading recall against the clutter inventory and a row corresponds to an IGNORE-surface path
|
|
- **THEN** that row is excluded from the recall calculation entirely (void), not counted as a miss
|
|
|
|
#### Scenario: A do-nothing pass cannot pass
|
|
|
|
- **WHEN** a calibration pass persists zero rules
|
|
- **THEN** it fails the recall floor and therefore cannot pass
|
|
|
|
### Requirement: Deterministic Nomination Intake Filter
|
|
|
|
Between cheap-model nomination and strong-model judgment, `:calibrate` SHALL
|
|
run a deterministic `NominationIntakeFilter` (in `calibrate_helpers.py`, no
|
|
model — invariant #6). A nomination whose glob+lifetime exactly equals a
|
|
`rejected` entry SHALL be dropped before the judge and logged in the run
|
|
summary. Surviving nominations SHALL be annotated with every related
|
|
rejection, where related means the two globs' match sets intersect on the
|
|
current shortlist (deterministic, computed from the scan). The annotations
|
|
plus all open consults SHALL enter the judge prompt as its "Nominations
|
|
memory" input section; related rejections are context for the judge, never a
|
|
veto.
|
|
|
|
#### Scenario: Exact glob+lifetime repeat is dropped before the judge
|
|
|
|
- **WHEN** haiku nominates `docs/research/** -> temporary` and `nominations.rejected` contains an entry with glob `docs/research/**` and lifetime `temporary`
|
|
- **THEN** the nomination is dropped at intake, never reaches the judge, and the drop is logged in the run summary
|
|
|
|
#### Scenario: A variant flows through annotated, not blocked
|
|
|
|
- **WHEN** haiku nominates `docs/research/drafts/** -> temporary` and `nominations.rejected` contains `docs/research/** -> temporary`, and the two globs' match sets intersect on the current shortlist
|
|
- **THEN** the nomination proceeds to the judge carrying the related rejection (its why and consider_instead) as context, and the judge may still confirm it
|
|
|
|
#### Scenario: Open consults always reach the judge prompt
|
|
|
|
- **WHEN** `nominations.consults` is non-empty at intake time
|
|
- **THEN** every open consult is included in the judge prompt's "Nominations memory" section, regardless of what haiku nominated this round
|
|
|
|
### Requirement: Consult Persistence and Resurfacing
|
|
|
|
Open `consult` verdicts SHALL persist to `nominations.consults` (deduped by
|
|
glob at write time) rather than dying with the run. Consults SHALL resurface
|
|
in `:calibrate` only — `:check` and `:clean` are unchanged — appearing in the
|
|
judge prompt and as an "Open consults" section of the rule report. A consult
|
|
SHALL exit in exactly one of three ways: (a) a human answer settles the
|
|
purpose — a normal rule is persisted and the consult entry deleted; (b) the
|
|
human deems it not rule-worthy — the entry is rewritten into `rejected` with
|
|
`rejected_by: "human"` and the human's why; (c) the human defers — the entry
|
|
stays and resurfaces next run. New rejections and consults SHALL appear in
|
|
the rule report but are not individually gated — they are memory, not
|
|
deletion authority.
|
|
|
|
#### Scenario: A consult survives the run and resurfaces
|
|
|
|
- **WHEN** a judge verdict is `consult` and the run ends without a human answer
|
|
- **THEN** the consult is written to `nominations.consults`, and the next `:calibrate` run surfaces it in both the judge prompt and the report's "Open consults" section
|
|
|
|
#### Scenario: An answered consult becomes a rule and disappears
|
|
|
|
- **WHEN** the human answers an open consult in a way that settles the artifact's purpose
|
|
- **THEN** a normal rule is persisted through the standard report flow and the consult entry is deleted in the same write
|
|
|
|
#### Scenario: A declined consult becomes a human rejection
|
|
|
|
- **WHEN** the human answers that an open consult's artifact class is not rule-worthy
|
|
- **THEN** the consult entry is rewritten into `nominations.rejected` with `rejected_by: "human"` and the stated reason
|
|
|
|
#### Scenario: Consults never surface outside calibrate
|
|
|
|
- **WHEN** `:check` or `:clean` runs against a project with open consults
|
|
- **THEN** their behavior is unchanged — consults neither appear in output nor affect any classification
|