cc-os/plugins/os-doc-hygiene/openspec/changes/lifecycle-aware-doc-hygiene/design.md

Design: lifecycle-aware-doc-hygiene

Context

os-doc-hygiene today only monitors stale/bloated docs (doc-check, doc-clean, report-schema specs, already shipped). The locked lifecycle design (plugins/os-doc-hygiene/lifecycle-spec.md, wayfinder map #31, ADRs 0038–0041) extends it to lifecycle management: every managed file gets a lifetime (keep / temporary / delete-once-served, plus an extract modifier), rules live in a rulebook, and a new :calibrate skill learns rules per project. This design translates the locked spec into the concrete seams of the existing pipeline (scanner.py → subagent classification → report_builder.py → validate_report.py → patch_applier.py) without reopening any decision already recorded in the spec or its ADRs.

Goals / Non-Goals

Goals

• Add a stdlib-only rulebook.py loader (global + per-project override, add-only merge, glob.translate dialect) consumed by the scanner.
• Give the scanner a lifecycle signal class where a directory-rule match prunes the walk — this is simultaneously the IGNORE-surface implementation (no separate ignore mechanism).
• Extend report_builder.py / validate_report.py / patch_applier.py to carry delete and extract-then-delete op types end-to-end under the ADR-0039 autonomy tier matrix, with all tracked/dirty state verified at application time via git, never trusted from the rule or the report cache.
• Implement temporary-tier age computation (git commit time, mtime fallback for untracked files, directory-inode mtime for untracked directory entries) and retain-recent-N semantics.
• Add conventions.json and wire promotion-candidate nudging into :check.
• Add :calibrate as a skill orchestrating haiku nomination + strong-model judge subagents, per the 6-step protocol in spec §8.
• Resolve the state-dir wrinkle (below) so the shipped IGNORE surface actually excludes the real state directory.

Non-Goals

• No ignore-surface propagation into .graphifyignore, .dochygiene-ignore, or any other tool's config (ADR-0040 — closed, not revisited here).
• No bulk-clean of any project other than cc-os (calibration pass #1 target per spec §9); no other project's rulebook is touched by this change.
• No recurring cross-project categorize-and-learn skill — that is charted as fog on map #31 and explicitly out of scope for this design (spec, "Out of scope" section).
• No propagate_ignore field, in any form (removed entirely per ADR-0040, not left as a documented-but-unused slot).
• No new deletion destinations beyond the existing knowledge-routing targets (ADR/CLAUDE.md/docs for repo-durable extraction, SecondBrain vault via /os-vault:write for cross-repo extraction).

Decisions

1. rulebook.py as a new stdlib-only module

A new scripts/rulebook.py, following the existing scripts' conventions (small single-responsibility classes, structured JSON-serializable output, no model, injectable for testing). Responsibilities:

• Load and parse the global plugins/os-doc-hygiene/rulebook.json and, if present, the project's committed .dochygiene-rules.json, both under the envelope {"schema_version": 1, "rules": [...]}.
• Compile every rule's glob once at load time via stdlib glob.translate(pattern, recursive=True, include_hidden=True) (Python ≥ 3.13, per spec §2) into a matchable regex; this is a hard runtime requirement on the Python version the plugin scripts run under — no fallback dialect is provided (matches the spec's explicit dialect pin).
• Merge add-only: project rules are appended to global rules, never replacing or deleting a global entry. A project "neutralizes" a global rule only by adding a shadowing rule with lifetime: "keep" at equal-or-higher precedence (per the two-axis precedence below) — there is no rule-removal mechanism, by design (ADR-0038).
• Resolve precedence per path: project file-rule > project directory-rule > global file-rule > global directory-rule; ties within the same tier broken by longest glob string, then by last-defined (definition order within the rules array, project-before-global order does not matter once the four buckets above are checked first).
• Validate: skip-and-warn per invalid or unconfirmed rule (a rule missing confirmed_by is loaded but flagged inactive, never contributes a signal, and never blocks the rest of the rulebook from loading). Hard-fail (raise / non-zero exit) only on unparseable JSON or an unrecognized schema_version.
• Expose a single query surface consumed by the scanner: given a project-root-relative path, return either None (unmatched — flows through existing signals unchanged) or the winning rule (with its precedence-resolved fields) plus whether the match was file-level or directory-level.

rulebook.py has no knowledge of git, classification, or the report schema — it only resolves "which rule, if any, governs this path," keeping it testable in complete isolation with fixture rulebook JSON and injected path lists, in line with every other script in scripts/.

2. Where the lifecycle signal plugs into scanner.py

A new signal class, LifecycleSignal (or equivalent), is attached during the existing scanner walk, alongside the current objective signals (broken refs, version skew, etc.). Concretely:

• Before recursing into a directory, the scanner asks the loaded rulebook whether a directory-rule matches that directory's path. If so, the walk is pruned — the scanner never opens any file beneath it — and the scanner emits one aggregate shortlist/signal entry for the directory path itself (not one entry per file inside it), carrying the lifecycle signal (rule ref, lifetime, served_when/served_when_path if present).
• This directory-rule-prune mechanism is the IGNORE surface described in spec §1 and §2 — there is no separate ignore-list data structure in the scanner. The IGNORE surface's seed members (graphify-out/**, .dochygiene/**) are shipped as ordinary directory rules with lifetime: "keep"... except keep implies "scanned and reported," which contradicts "never walked." Resolution: IGNORE-surface entries are a rulebook rule with no lifetime field at all (or a reserved sentinel the scanner recognizes before even considering lifetime) whose sole effect is walk-pruning with no shortlist/signal emission at all — distinct from keep, which is walked and reported. See "Ambiguities resolved" below; this is the one place the spec's wording ("directory-rule match prunes the walk and emits one aggregate entry — this IS the ignore surface") needed disambiguating against the separate statement that IGNORE-surface members are "never walked" with no entry at all. This design follows the latter (spec §1, §3 table: "IGNORE surface … never walked", no aggregate-entry language attached to it) and reserves the "prunes the walk and emits one aggregate entry" behavior for ordinary temporary/delete-once-served directory rules (e.g., autoresearch/<run-id>/), which need a signal to drive deletion decisions, whereas pure IGNORE members need none.
• For files matched by a file-rule (not caught by a directory prune), the scanner attaches the lifecycle signal to that file's existing entry in the shortlist, alongside any pre-existing objective signals — a file can be both lifecycle-tagged and, say, stale-by-broken-ref.
• The lifecycle signal is consumed downstream by the classification subagent as a new signal class (per doc-check's existing "signals are passed through verbatim" contract) and ultimately drives op/op_type selection (delete / extract-then-delete) in place of, or alongside, the existing stale/bloat vocabulary.

3. Delete / extract-then-delete op flow

report_builder.py → validate_report.py → patch_applier.py, following the existing division of labor (model proposes judgment fields only; deterministic finalize pass authors the guardrail fields; validator enforces the schema; applier mutates at clean time with runtime re-verification).

• report_builder.py gains two new exact_edit.kind values in its KIND_TABLE: delete and extract-then-delete. Both carry an anchor covering the full file (or, for a directory-rule aggregate entry, the directory path with no anchor) so the biconditional with op_type holds. extract-then-delete additionally carries the extraction destination classification (repo-durable vs cross-repo) as a proposal field the model supplies (it is a judgment about content, not a derived guardrail field) plus, for repo-durable, a target doc reference; cross-repo routes through /os-vault:write at clean time and carries no fixed destination path (spec §1: "no new destinations").
• validate_report.py's derive_safety_tier remains the single source of tier derivation (invariant #10) and is extended, not replaced, with the ADR-0039 matrix as additional input dimensions beyond (op_type, is_destructive, is_reversible):
  • a lifecycle object on the entry (rule_ref, lifetime, served_when_path XOR served_when, git_state placeholder recomputed at runtime — see below)
  • derive_safety_tier gains a lifecycle-aware branch: for op_type in (delete, extract-then-delete), tier is auto only when the entry's lifecycle evidence is scanner-proven (a served_when_path hit, or a temporary-tier age/retain-recent computation) and the file's git state is tracked+clean; every other combination (dirty, untracked, or served_when classifier-judged) forces confirm. This is additive: the existing non-lifecycle branches (stale/bloat ops) are unchanged, and derive_safety_tier is still the one function both report_builder.py and validate_report.py call — no second tier-deriving code path is introduced.
  • Because tracked/dirty state can only be known against the live worktree, report_builder.py calls a runtime git ls-files + dirty check at report-build time to populate the git_state input for tier derivation — but per ADR-0039 this is explicitly re-verified again at apply time by patch_applier.py, because time passes between check and clean. The report's tier is advisory-fresh, not authoritative-forever.
• patch_applier.py gains handling for delete and extract-then-delete:
  • Before applying either, it re-runs git ls-files <path> and a dirty check against that specific path (never trusting the report's cached tier or git_state) — if the path is no longer tracked+clean when the rule required that for auto, the applier downgrades that entry to skip-and-report (git-state-changed-since-check), the same family of guard as the existing content-hash guard, not a silent auto-apply.
  • delete performs a true git rm <path> (or git rm -r for a directory-rule aggregate entry) staged into the same single hygiene commit the rest of the run produces — no archive directory, no move. This is staged precisely like every other op (no git add -A).
  • extract-then-delete first invokes the generative extraction (repo- durable → written via the same live-read Sonnet-subagent path doc-clean already uses for generative ops, target = ADR/CLAUDE.md/docs per the model's proposal; cross-repo → /os-vault:write invoked by the clean skill, not the applier itself, since vault writes are not a filesystem op the applier owns) and only after that step succeeds does it git rm the source path, both landing in the one hygiene commit. If extraction fails, the delete does not proceed for that entry (fails closed, consistent with the existing "hard failure → rollback, partial-guard-skip → no rollback" split: an extraction failure is a per-entry skip, not a run-level hard failure, unless it is the kind of error doc-clean's existing rollback rules already treat as hard).

4. Temporary-tier age computation

• Age source: git commit time of the path's most recent commit touching it, obtained via git log -1 --format=%cI -- <path>; falls back to filesystem mtime only when the path is untracked (no commit history to read). No per-rule age_source override field exists (ADR-0039/#48).
• Untracked directory entries (a directory-rule match on an untracked directory) use one stat() on the directory inode itself, not a recursive max-mtime walk over its contents — cheap and self-healing per spec §4/§48 (a spuriously bumped directory mtime only delays deletion by one round, it never causes incorrect early deletion).
• Retention unit is the rule's own match granularity: a file-rule's matches are individual files; a directory-rule's matches are whole directories (e.g. autoresearch/<run-id>/ — "3 most recent runs," not "3 most recent files inside the newest run"). rulebook.py and the scanner must agree on this: the directory-rule aggregate shortlist entry (§2 above) is the unit retain_recent/max_age_days operate over, computed by grouping sibling directory-rule matches under their common parent glob and ranking by age (newest first), keeping the top retain_recent regardless of age and deleting the rest once they exceed max_age_days.
• retain_recent defaults to 3, max_age_days defaults to 3, both per-rule overridable in the rulebook JSON.

5. conventions.json and promotion candidates

• plugins/os-doc-hygiene/conventions.json is a global-only, plugin-shipped, machine-readable file (envelope TBD-but-simple: an array of convention objects — name, "what it proves," served_when_path/frontmatter template a rule graduates to, one-line human pitch). v1 ships exactly the two entries named in the spec (archive-bucket, status-frontmatter); no per-project override file exists for it (ADR-0041: "the catalog only recommends; adoption lands in the project's own rulebook"). report_builder.py (deterministic, no model) reads this file directly — no subagent involvement — and, for every entry whose lifecycle signal is classifier-judged (served_when present, no served_when_path), checks whether any catalog convention's served_when_path pattern is not yet satisfied by that project's current structure, and if so emits a promotion_candidates entry naming the convention and the one-line pitch. This is a deterministic string/structure check, not a judgment call, so it belongs in report_builder.py alongside the other guardrail fields the model must not author.
• The :check report gains a promotion_candidates array section at the top level (sibling to entries), populated on every run where at least one classifier-judged lifecycle signal exists and a matching catalog convention applies. :calibrate (out of this change's core report path, but sharing the same catalog) may go further and draft the adoption (graduated rule + the file moves that convention implies) for human approval, never applying unasked.

6. :calibrate as an orchestrating skill

New skills/calibrate/SKILL.md (the only new skill this change adds), following the existing skill pattern (check/clean/sweep SKILL.md + a workflows/*.md LOOP-GUARD subagent pointer where a subagent must not recurse into the top-level skill file). It orchestrates, per spec §8:

1. A deterministic clustering pass (script, no model) over the current scanner's unmatched shortlist (unmatched = unmanaged = candidate pool), grouping by path-shape similarity and sampling representatives per cluster — this is a new small deterministic helper, not a new pipeline stage in check/clean, since calibration is a separate on-demand skill.
2. A haiku subagent per cluster, constrained by prompt contract to nominate a bare glob + candidate lifetime, never an exact-instance path (the rule-quality "class, never path" test is enforced by the strong- model judge in the next step, not trusted from haiku).
3. One batched strong-model (Opus/Fable) judge subagent that gathers its own evidence (re-reads matched paths, checks near-miss boundaries) and returns verdicts (confirm / reject / amend / consult), with consult mandatory whenever an artifact's purpose is unclear.
4. A deterministic report-assembly step (script) that renders the required 5-element rule report (glob verbatim, matches with capped sample + total count, near-miss boundary, tier, plain-language why) to the human before any persistence call is made — persistence is a separate, explicit step gated on human review of this report, not something the judge subagent triggers directly.
5. Persistence: project-rule writes to .dochygiene-rules.json happen on judge confirmation (post human rule-report review); global-rulebook writes are additionally human-gated (a distinct confirmation, since it is a cross-repo write into cc-os); rule removals are HITL-only in all cases.
6. A retest loop (re-run clustering against the shrunk unmatched pool) that stops when a round nominates fewer than 2 new rules or shrinks the unmatched pool by less than 10%, hard-capped at 3 rounds regardless.

7. The state-dir wrinkle

lifecycle-spec.md's IGNORE seed lists .dochygiene/**. Since ADR-0027 (pre-dating this change), the plugin's actual state directory is .cc-os/dochygiene/, with .dochygiene/ retained only as a legacy read- fallback that state_store.py migrates away from on first write; scanner.py already self-excludes both .cc-os/ and .dochygiene/ by directory name at any depth (per scripts/CONTEXT.md's documented default excludes). This change's rulebook-driven IGNORE surface must therefore ship both seed entries — .dochygiene/** (matching the spec text verbatim, covering the legacy dir on projects that haven't migrated) and an equivalent rule (or reliance on the scanner's pre-existing hardcoded self-exclusion, which already covers .cc-os/**) — so that on a project already migrated to .cc-os/dochygiene/, the IGNORE surface still holds. This design does not change the scanner's existing hardcoded self-exclusion; the rulebook's IGNORE rule list is additive to it, not a replacement, so this is satisfied without a code change beyond confirming the shipped rulebook.json's IGNORE entries include .dochygiene/** and documenting that .cc-os/** is already covered by the pre-existing self-exclusion.

Risks / Trade-offs

• Two overlapping walk-pruning mechanisms (the scanner's pre-existing hardcoded excluded_dirs and the new rulebook-driven directory-rule prune) risk confusing future maintainers about which one is authoritative for a given path. Mitigated by treating the hardcoded excludes as a distinct, lower-level safety net (self-exclusion of the tool's own state and pre-existing fixture excludes) and the rulebook as the project-configurable lifecycle layer; scripts/CONTEXT.md should document both once implemented.
• git ls-files / dirty check runs twice (once at report-build time, once at apply time) for lifecycle deletes — a deliberate, spec-mandated redundancy (ADR-0039: "never trusting the rule … at runtime") rather than an oversight; the cost is one extra git subprocess call per lifecycle entry per run, acceptable given the destructive nature of the operation.
• extract-then-delete spans two subsystems (a generative rewrite via the existing Sonnet-distillation path, then a git rm) inside a single op type, and for cross-repo extraction additionally calls into /os-vault:write, a different plugin's skill. This is more moving parts in one applied entry than any existing op; a partial failure (extraction succeeds, delete fails, or vice versa) needs care to keep the "one hygiene commit" invariant intact — this design treats extraction-then-delete as atomic per entry (both steps land in the same commit, or neither does), which may need revisiting once implemented against real vault-write latency/failure modes.
• Directory-rule aggregate entries change the meaning of "one entry per file" that today's doc-check/report-schema specs assume throughout (e.g. "every entries[].path SHALL be a member of shortlist" — a directory aggregate entry's path is a directory, not a file, which is a new shape for path that existing consumers may not expect). The delta specs below extend rather than replace this requirement, but any code outside this change's touched files that assumes path is always a regular file should be audited during implementation.
• rulebook.py's hard requirement on Python ≥3.13 (glob.translate) is new relative to the rest of the scripts directory (whose stdlib-only policy has not previously pinned a Python floor this high); if the environment running os-doc-hygiene scripts is older, the whole rulebook layer hard-fails to import. This is accepted as a locked design decision (spec §2), not reopened here, but is called out as an operational risk worth a version check with a clear error message at load time.

Ambiguities resolved while writing this design

(Reported per instructions — none silently decided against the spec; these are places the locked documents left an implementation-level gap.)

1. IGNORE-surface entries vs. ordinary directory-rule "prune + aggregate entry" behavior. Spec §1/§2 phrasing could be read as saying the IGNORE surface itself produces an aggregate shortlist entry ("a directory-rule match prunes the walk and emits one aggregate entry — this IS the ignore surface"), but spec §1 and the §3 tier table also say IGNORE-surface members are simply "never walked," with no entry language. Resolved by treating "prune + aggregate entry" as the general mechanism for lifecycle directory rules with a real lifetime (temporary / delete-once-served), and reserving true zero-signal, zero-entry pruning for the specific IGNORE seed list (graphify-out/**, .dochygiene/**). Flagging this because it affects whether IGNORE members ever appear anywhere in a report — this design says they never do.
2. conventions.json file shape. The spec names the file and its v1 contents (archive-bucket, status-frontmatter) and per-entry conceptual fields (name, what it proves, template, pitch) but does not give a JSON schema. This design treats it as a flat array of objects with those four conceptual fields, no envelope/schema_version wrapper (unlike rulebook.json, which the spec explicitly gives an envelope). Flagging because a schema_version envelope could be added instead if implementation prefers consistency with rulebook.json; nothing in the spec forecloses either choice.
3. Where the git ls-files/dirty runtime check physically lives (spec says "verified at runtime via git ls-files + dirty check, never trusting the rule" but doesn't say whether that's a report-build-time check, an apply-time check, or both). This design does both — advisory at report-build time (so the report's tier reflects current reality when shown to the human), authoritative at apply time (so a stale report never causes an unsafe auto-delete) — matching the existing content-hash-guard pattern (expected_sha256 is likewise computed at build time and re-verified at apply time). Flagging because the spec's single sentence doesn't explicitly mandate the double-check; it was inferred from ADR-0039's "never trusting the rule" plus the existing applier's re-verification precedent (patch_applier.py's content-hash guard).
4. Extraction-failure handling within extract-then-delete (does a failed extraction hard-fail the whole clean run, or just skip that entry?) is not addressed in the spec. This design treats it as a per-entry skip (consistent with doc-clean's existing partial-success semantics), not a run-level hard failure, unless the failure mode matches one of the existing hard-failure triggers (applier exit 2, write error). Flagging since this determines commit granularity under failure.