144 lines
7.6 KiB
Markdown
144 lines
7.6 KiB
Markdown
# Design: calibrate-assessment-inventory
|
|
|
|
## Context
|
|
|
|
Wayfinder map #49 produced a fully locked design, recorded in
|
|
`plugins/os-doc-hygiene/lifecycle-spec.md` (§1 extract index, §2 nominations
|
|
memory + canonical ordering, §8 protocol steps 3/6 + test coverage) and the
|
|
2026-07-15 amendments to ADR-0038/0039, committed as `9f11df3`. This change
|
|
implements it. There are no open design questions — where this document
|
|
states a choice, it is restating the locked decision and its rationale, not
|
|
making a new one.
|
|
|
|
Current state: `:calibrate` persists only confirmed rules; keep verdicts,
|
|
human declines, and open consults evaporate with the run's scratch dir and
|
|
are re-derived every pass. `calibrate_helpers.py` has helper functions but no
|
|
intake filter and no single serialization path for `.dochygiene-rules.json`.
|
|
The `:clean` extract-then-delete op writes to the vault but leaves no in-repo
|
|
pointer.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- Persist every settled judge verdict: answers as plain rules (keep included),
|
|
"no" as `nominations.rejected`, "not yet" as `nominations.consults`.
|
|
- Deterministically block exact-repeat re-nominations and hand the judge its
|
|
history (related rejections + open consults) as context.
|
|
- One canonical writer for `.dochygiene-rules.json` so every write is
|
|
ordering-stable and reviewable.
|
|
- Leave a discoverable pointer (`extracted.md`) when an extraction physically
|
|
leaves the repo for the vault.
|
|
|
|
**Non-Goals:**
|
|
|
|
- No changes to `rulebook.py` semantics — it stays nomination-unaware
|
|
(contract: "which rule governs this path"). Only `calibrate_helpers` reads
|
|
the `nominations` key.
|
|
- No consult resurfacing in `:check`/`:clean` (ADR-0039 boundary — calibrate
|
|
only).
|
|
- No automated revisit of rejections or singleton keep rules — both exit by
|
|
hand-deletion only.
|
|
- No new eval harness for the judge-context / consult-loop behavior (#59) —
|
|
worked examples + production IRL audits instead.
|
|
- No post-write hook for ordering enforcement (rejected in #53).
|
|
|
|
## Decisions
|
|
|
|
All decided on map #49; restated with rationale:
|
|
|
|
1. **Keep verdicts are plain `lifetime: keep` rules, not a separate store**
|
|
(#51). Matched = managed removes them from the calibrate pool and the glob
|
|
protects future files for free. Corollary: the class-never-path test is
|
|
relaxed for the keep tier only — a singleton keep merely protects, so
|
|
exact-path globs are allowed there and remain forbidden for
|
|
`temporary`/`delete-once-served`.
|
|
2. **`nominations` holds only the two non-rule residues** (#52): `rejected`
|
|
("the answer is no") and `consults` ("no answer yet"). It never touches the
|
|
file filter. Entry schemas per lifecycle-spec §2; consults deliberately
|
|
carry no lifetime; presence in `consults` = open, no status field.
|
|
3. **Rejections block exact glob+lifetime repeats only** (#52). Variants flow
|
|
to the judge annotated with the related rejection — memory, not veto.
|
|
Relatedness is deterministic: the two globs' match sets intersect on the
|
|
current shortlist.
|
|
4. **The intake filter is deterministic code, not a model step** (#52,
|
|
invariant #6): `NominationIntakeFilter` in `calibrate_helpers.py`, run as
|
|
pipeline Step 3.5 between haiku nomination and the judge; drops are logged
|
|
in the run summary.
|
|
5. **Canonical ordering is writer-enforced, no hook** (#53). Grouping:
|
|
`rules` by tier (delete-once-served, temporary, keep), glob-sorted within
|
|
group; `nominations` after `rules`; `consults` before `rejected` (pending
|
|
-action queue reads first), each glob-sorted. Idempotent; hand edits
|
|
re-canonicalize on the next write. Unknown fields warn and round-trip
|
|
(never dropped).
|
|
6. **Consult exits** (#56): (a) human answer settles purpose → normal rule
|
|
persisted, consult entry deleted; (b) not rule-worthy → rewritten into
|
|
`rejected` with `rejected_by: "human"`; (c) defer → stays, resurfaces next
|
|
run. New rejections/consults appear in the Step 5 report but are not
|
|
individually gated — they are memory, not deletion authority.
|
|
7. **`extracted.md` append is owned by the extract op** (#57): distill →
|
|
`/os-vault:write` → append pointer line → `git rm`, all inside the same
|
|
per-file transaction and single hygiene commit — no window where the doc is
|
|
gone but undiscoverable. Vault extractions only; repo-durable residue
|
|
(ADR/CLAUDE.md) gets no index line. Self-protection is the already-shipped
|
|
global `**/extracted.md → keep` rule.
|
|
8. **Test coverage** (#59): unit tests only, for the two deterministic seams
|
|
(intake filter; writer). The two model-facing behaviors get worked examples
|
|
(already landed in judge.md / SKILL.md with the design commit) and IRL
|
|
audits as the next signal.
|
|
|
|
## Implementation shape
|
|
|
|
- `scripts/calibrate_helpers.py`: add `NominationIntakeFilter` (small
|
|
single-responsibility class, injected shortlist/rules-file data, returns
|
|
`(survivors_with_annotations, dropped_log)`) and a `RulesFileWriter` (or
|
|
extend the existing write path) that owns *all* serialization of
|
|
`.dochygiene-rules.json`, including nominations read/validate/warn.
|
|
- `skills/calibrate/SKILL.md` + workflow files: wire Step 3.5 (run the filter,
|
|
feed its annotations + open consults into the judge prompt's existing
|
|
"Nominations memory" section), Step 5 ("Open consults" report section,
|
|
consult exits a/b/c), Step 6 (persist keep verdicts / rejections / consults
|
|
through the canonical writer, dedupe consults by glob at write time). The
|
|
SKILL.md consult-loop worked example loses its "design-level pending
|
|
wiring" marker.
|
|
- `scripts/patch_applier.py`: extend the `extract-then-delete` path — when the
|
|
extraction destination is the vault, append the pointer entry (format per
|
|
lifecycle-spec §1) to `<dir>/extracted.md` (create if absent) and stage it
|
|
in the same hygiene commit; on extraction failure, neither append nor
|
|
delete applies (existing skip semantics).
|
|
- `tests/`: extend `test_calibrate_helpers.py` (or add
|
|
`test_nomination_intake.py` / writer tests) covering exact-repeat drops,
|
|
the two #52 worked relatedness cases, round-trip canonical ordering, and
|
|
unknown-field warnings; patch-applier tests cover the append-then-delete
|
|
atomicity and the no-append-on-repo-durable-extraction case.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- [Nominations grow without bound — stale rejections accumulate] → accepted
|
|
by design: exits are hand-deletion only (removals stay HITL); the canonical
|
|
ordering keeps the file reviewable, and entries are small.
|
|
- [Judge treats a related rejection as a veto and under-nominates] → the
|
|
judge.md "Nominations memory" section states variants-are-not-vetoes with a
|
|
worked example; IRL session audits are the detection path (#59), and IRL
|
|
evidence triggers harness design per eval discipline.
|
|
- [Writer round-trips unknown fields but a hand-edit introduces malformed
|
|
JSON] → existing skip-and-warn loader discipline applies; the writer only
|
|
runs on an already-parsed structure.
|
|
- [extracted.md append fails after a successful vault write] → the per-file
|
|
transaction skips the delete too, so the doc is never gone-but-unindexed;
|
|
the vault note is orphaned-but-harmless and the next run retries.
|
|
|
|
## Migration Plan
|
|
|
|
No data migration: `nominations` is additive to `schema_version` 1 and the v1
|
|
loader ignores unknown top-level keys. Existing `.dochygiene-rules.json`
|
|
files re-canonicalize on their first post-change write. Rollback is code-only
|
|
(revert the change; `nominations` keys left in project files are inert).
|
|
After implementation: full test suite green (407 existing tests + new), then
|
|
`bin/refresh-plugins`.
|
|
|
|
## Open Questions
|
|
|
|
None — the design is locked (map #49); ambiguities found during
|
|
implementation go back to `lifecycle-spec.md` as the source of truth.
|