# Design: lifecycle-aware-doc-hygiene ## Context os-doc-hygiene today only monitors stale/bloated docs (`doc-check`, `doc-clean`, `report-schema` specs, already shipped). The locked lifecycle design (`plugins/os-doc-hygiene/lifecycle-spec.md`, wayfinder map #31, ADRs 0038–0041) extends it to *lifecycle management*: every managed file gets a lifetime (`keep` / `temporary` / `delete-once-served`, plus an `extract` modifier), rules live in a rulebook, and a new `:calibrate` skill learns rules per project. This design translates the locked spec into the concrete seams of the existing pipeline (`scanner.py` → subagent classification → `report_builder.py` → `validate_report.py` → `patch_applier.py`) without reopening any decision already recorded in the spec or its ADRs. ## Goals / Non-Goals **Goals** - Add a stdlib-only `rulebook.py` loader (global + per-project override, add-only merge, `glob.translate` dialect) consumed by the scanner. - Give the scanner a lifecycle signal class where a directory-rule match prunes the walk — this is simultaneously the IGNORE-surface implementation (no separate ignore mechanism). - Extend `report_builder.py` / `validate_report.py` / `patch_applier.py` to carry `delete` and `extract-then-delete` op types end-to-end under the ADR-0039 autonomy tier matrix, with all tracked/dirty state verified at application time via git, never trusted from the rule or the report cache. - Implement temporary-tier age computation (git commit time, mtime fallback for untracked files, directory-inode mtime for untracked directory entries) and retain-recent-N semantics. - Add `conventions.json` and wire promotion-candidate nudging into `:check`. - Add `:calibrate` as a skill orchestrating haiku nomination + strong-model judge subagents, per the 6-step protocol in spec §8. - Resolve the state-dir wrinkle (below) so the shipped IGNORE surface actually excludes the real state directory. **Non-Goals** - No ignore-surface propagation into `.graphifyignore`, `.dochygiene-ignore`, or any other tool's config (ADR-0040 — closed, not revisited here). - No bulk-clean of any project other than cc-os (calibration pass #1 target per spec §9); no other project's rulebook is touched by this change. - No recurring cross-project categorize-and-learn skill — that is charted as fog on map #31 and explicitly out of scope for this design (spec, "Out of scope" section). - No `propagate_ignore` field, in any form (removed entirely per ADR-0040, not left as a documented-but-unused slot). - No new deletion destinations beyond the existing knowledge-routing targets (ADR/CLAUDE.md/docs for repo-durable extraction, SecondBrain vault via `/os-vault:write` for cross-repo extraction). ## Decisions ### 1. `rulebook.py` as a new stdlib-only module A new `scripts/rulebook.py`, following the existing scripts' conventions (small single-responsibility classes, structured JSON-serializable output, no model, injectable for testing). Responsibilities: - Load and parse the global `plugins/os-doc-hygiene/rulebook.json` and, if present, the project's committed `.dochygiene-rules.json`, both under the envelope `{"schema_version": 1, "rules": [...]}`. - Compile every rule's `glob` once at load time via stdlib `glob.translate(pattern, recursive=True, include_hidden=True)` (Python ≥ 3.13, per spec §2) into a matchable regex; this is a hard runtime requirement on the Python version the plugin scripts run under — no fallback dialect is provided (matches the spec's explicit dialect pin). - Merge add-only: project rules are appended to global rules, never replacing or deleting a global entry. A project "neutralizes" a global rule only by adding a shadowing rule with `lifetime: "keep"` at equal-or-higher precedence (per the two-axis precedence below) — there is no rule-removal mechanism, by design (ADR-0038). - Resolve precedence per path: project file-rule > project directory-rule > global file-rule > global directory-rule; ties within the same tier broken by longest `glob` string, then by last-defined (definition order within the rules array, project-before-global order does not matter once the four buckets above are checked first). - Validate: skip-and-warn per invalid or unconfirmed rule (a rule missing `confirmed_by` is loaded but flagged `inactive`, never contributes a signal, and never blocks the rest of the rulebook from loading). Hard-fail (raise / non-zero exit) only on unparseable JSON or an unrecognized `schema_version`. - Expose a single query surface consumed by the scanner: given a project-root-relative path, return either `None` (unmatched — flows through existing signals unchanged) or the winning rule (with its precedence-resolved fields) plus whether the match was file-level or directory-level. `rulebook.py` has no knowledge of git, classification, or the report schema — it only resolves "which rule, if any, governs this path," keeping it testable in complete isolation with fixture rulebook JSON and injected path lists, in line with every other script in `scripts/`. ### 2. Where the lifecycle signal plugs into `scanner.py` A new signal class, `LifecycleSignal` (or equivalent), is attached during the existing scanner walk, alongside the current objective signals (broken refs, version skew, etc.). Concretely: - Before recursing into a directory, the scanner asks the loaded rulebook whether a directory-rule matches that directory's path. If so, the walk is **pruned** — the scanner never opens any file beneath it — and the scanner emits **one aggregate shortlist/signal entry** for the directory path itself (not one entry per file inside it), carrying the lifecycle signal (rule ref, lifetime, `served_when`/`served_when_path` if present). - This directory-rule-prune mechanism **is** the IGNORE surface described in spec §1 and §2 — there is no separate ignore-list data structure in the scanner. The IGNORE surface's seed members (`graphify-out/**`, `.dochygiene/**`) are shipped as ordinary directory rules with `lifetime: "keep"`... except `keep` implies "scanned and reported," which contradicts "never walked." Resolution: IGNORE-surface entries are a rulebook rule with no `lifetime` field at all (or a reserved sentinel the scanner recognizes before even considering lifetime) whose sole effect is walk-pruning with **no** shortlist/signal emission at all — distinct from `keep`, which is walked and reported. See "Ambiguities resolved" below; this is the one place the spec's wording ("directory-rule match prunes the walk and emits one aggregate entry — this IS the ignore surface") needed disambiguating against the separate statement that IGNORE-surface members are "never walked" with no entry at all. This design follows the latter (spec §1, §3 table: "IGNORE surface … never walked", no aggregate-entry language attached to it) and reserves the "prunes the walk and emits one aggregate entry" behavior for ordinary `temporary`/`delete-once-served` directory rules (e.g., `autoresearch//`), which need a signal to drive deletion decisions, whereas pure IGNORE members need none. - For files matched by a file-rule (not caught by a directory prune), the scanner attaches the lifecycle signal to that file's existing entry in the shortlist, alongside any pre-existing objective signals — a file can be both lifecycle-tagged and, say, stale-by-broken-ref. - The lifecycle signal is consumed downstream by the classification subagent as a new signal class (per doc-check's existing "signals are passed through verbatim" contract) and ultimately drives `op`/`op_type` selection (`delete` / `extract-then-delete`) in place of, or alongside, the existing stale/bloat vocabulary. ### 3. Delete / extract-then-delete op flow `report_builder.py` → `validate_report.py` → `patch_applier.py`, following the existing division of labor (model proposes judgment fields only; deterministic finalize pass authors the guardrail fields; validator enforces the schema; applier mutates at clean time with runtime re-verification). - **`report_builder.py`** gains two new `exact_edit.kind` values in its `KIND_TABLE`: `delete` and `extract-then-delete`. Both carry an `anchor` covering the full file (or, for a directory-rule aggregate entry, the directory path with no anchor) so the biconditional with `op_type` holds. `extract-then-delete` additionally carries the extraction destination classification (`repo-durable` vs `cross-repo`) as a proposal field the model supplies (it is a judgment about content, not a derived guardrail field) plus, for `repo-durable`, a target doc reference; `cross-repo` routes through `/os-vault:write` at clean time and carries no fixed destination path (spec §1: "no new destinations"). - **`validate_report.py`**'s `derive_safety_tier` remains the single source of tier derivation (invariant #10) and is extended, not replaced, with the ADR-0039 matrix as additional input dimensions beyond `(op_type, is_destructive, is_reversible)`: - a `lifecycle` object on the entry (`rule_ref`, `lifetime`, `served_when_path` XOR `served_when`, `git_state` placeholder recomputed at runtime — see below) - `derive_safety_tier` gains a lifecycle-aware branch: for `op_type` in (`delete`, `extract-then-delete`), tier is `auto` **only** when the entry's lifecycle evidence is `scanner-proven` (a `served_when_path` hit, or a temporary-tier age/retain-recent computation) **and** the file's git state is tracked+clean; every other combination (dirty, untracked, or `served_when` classifier-judged) forces `confirm`. This is additive: the existing non-lifecycle branches (stale/bloat ops) are unchanged, and `derive_safety_tier` is still the one function both `report_builder.py` and `validate_report.py` call — no second tier-deriving code path is introduced. - Because tracked/dirty state can only be known against the live worktree, `report_builder.py` calls a runtime `git ls-files` + dirty check **at report-build time** to populate the `git_state` input for tier derivation — but per ADR-0039 this is explicitly re-verified again at apply time by `patch_applier.py`, because time passes between check and clean. The report's tier is advisory-fresh, not authoritative-forever. - **`patch_applier.py`** gains handling for `delete` and `extract-then-delete`: - Before applying either, it re-runs `git ls-files ` and a dirty check against that specific path (never trusting the report's cached tier or git_state) — if the path is no longer tracked+clean when the rule required that for `auto`, the applier downgrades that entry to skip-and-report (`git-state-changed-since-check`), the same family of guard as the existing content-hash guard, not a silent auto-apply. - `delete` performs a true `git rm ` (or `git rm -r` for a directory-rule aggregate entry) staged into the same single hygiene commit the rest of the run produces — no archive directory, no move. This is staged precisely like every other op (no `git add -A`). - `extract-then-delete` first invokes the generative extraction (repo- durable → written via the same live-read Sonnet-subagent path `doc-clean` already uses for generative ops, target = ADR/CLAUDE.md/docs per the model's proposal; cross-repo → `/os-vault:write` invoked by the clean skill, not the applier itself, since vault writes are not a filesystem op the applier owns) and only after that step succeeds does it `git rm` the source path, both landing in the one hygiene commit. If extraction fails, the delete does not proceed for that entry (fails closed, consistent with the existing "hard failure → rollback, partial-guard-skip → no rollback" split: an extraction failure is a per-entry skip, not a run-level hard failure, unless it is the kind of error `doc-clean`'s existing rollback rules already treat as hard). ### 4. Temporary-tier age computation - **Age source:** git commit time of the path's most recent commit touching it, obtained via `git log -1 --format=%cI -- `; falls back to filesystem `mtime` only when the path is untracked (no commit history to read). No per-rule `age_source` override field exists (ADR-0039/#48). - **Untracked directory entries** (a directory-rule match on an untracked directory) use **one `stat()` on the directory inode itself**, not a recursive max-mtime walk over its contents — cheap and self-healing per spec §4/§48 (a spuriously bumped directory mtime only delays deletion by one round, it never causes incorrect *early* deletion). - **Retention unit** is the rule's own match granularity: a file-rule's matches are individual files; a directory-rule's matches are whole directories (e.g. `autoresearch//` — "3 most recent runs," not "3 most recent files inside the newest run"). `rulebook.py` and the scanner must agree on this: the directory-rule aggregate shortlist entry (§2 above) is the unit `retain_recent`/`max_age_days` operate over, computed by grouping sibling directory-rule matches under their common parent glob and ranking by age (newest first), keeping the top `retain_recent` regardless of age and deleting the rest once they exceed `max_age_days`. - `retain_recent` defaults to 3, `max_age_days` defaults to 3, both per-rule overridable in the rulebook JSON. ### 5. `conventions.json` and promotion candidates - `plugins/os-doc-hygiene/conventions.json` is a global-only, plugin-shipped, machine-readable file (envelope TBD-but-simple: an array of convention objects — name, "what it proves," `served_when_path`/frontmatter template a rule graduates to, one-line human pitch). v1 ships exactly the two entries named in the spec (`archive-bucket`, `status-frontmatter`); no per-project override file exists for it (ADR-0041: "the catalog only recommends; adoption lands in the project's own rulebook"). `report_builder.py` (deterministic, no model) reads this file directly — no subagent involvement — and, for every entry whose lifecycle signal is classifier-judged (`served_when` present, no `served_when_path`), checks whether any catalog convention's `served_when_path` pattern is *not yet* satisfied by that project's current structure, and if so emits a `promotion_candidates` entry naming the convention and the one-line pitch. This is a deterministic string/structure check, not a judgment call, so it belongs in `report_builder.py` alongside the other guardrail fields the model must not author. - The `:check` report gains a `promotion_candidates` array section at the top level (sibling to `entries`), populated on every run where at least one classifier-judged lifecycle signal exists and a matching catalog convention applies. `:calibrate` (out of this change's core report path, but sharing the same catalog) may go further and draft the adoption (graduated rule + the file moves that convention implies) for human approval, never applying unasked. ### 6. `:calibrate` as an orchestrating skill New `skills/calibrate/SKILL.md` (the only new skill this change adds), following the existing skill pattern (`check`/`clean`/`sweep` SKILL.md + a `workflows/*.md` LOOP-GUARD subagent pointer where a subagent must not recurse into the top-level skill file). It orchestrates, per spec §8: 1. A deterministic clustering pass (script, no model) over the current scanner's unmatched shortlist (unmatched = unmanaged = candidate pool), grouping by path-shape similarity and sampling representatives per cluster — this is a new small deterministic helper, not a new pipeline stage in `check`/`clean`, since calibration is a separate on-demand skill. 2. A **haiku** subagent per cluster, constrained by prompt contract to nominate a bare glob + candidate lifetime, never an exact-instance path (the rule-quality "class, never path" test is enforced by the strong- model judge in the next step, not trusted from haiku). 3. One batched **strong-model** (Opus/Fable) judge subagent that gathers its own evidence (re-reads matched paths, checks near-miss boundaries) and returns verdicts (`confirm` / `reject` / `amend` / `consult`), with `consult` mandatory whenever an artifact's purpose is unclear. 4. A deterministic report-assembly step (script) that renders the required 5-element rule report (glob verbatim, matches with capped sample + total count, near-miss boundary, tier, plain-language why) to the human **before any persistence call is made** — persistence is a separate, explicit step gated on human review of this report, not something the judge subagent triggers directly. 5. Persistence: project-rule writes to `.dochygiene-rules.json` happen on judge confirmation (post human rule-report review); global-rulebook writes are additionally human-gated (a distinct confirmation, since it is a cross-repo write into cc-os); rule removals are HITL-only in all cases. 6. A retest loop (re-run clustering against the shrunk unmatched pool) that stops when a round nominates fewer than 2 new rules or shrinks the unmatched pool by less than 10%, hard-capped at 3 rounds regardless. ### 7. The state-dir wrinkle `lifecycle-spec.md`'s IGNORE seed lists `.dochygiene/**`. Since ADR-0027 (pre-dating this change), the plugin's actual state directory is `.cc-os/dochygiene/`, with `.dochygiene/` retained only as a legacy read- fallback that `state_store.py` migrates away from on first write; `scanner.py` already self-excludes both `.cc-os/` and `.dochygiene/` by directory name at any depth (per `scripts/CONTEXT.md`'s documented default excludes). This change's rulebook-driven IGNORE surface must therefore ship **both** seed entries — `.dochygiene/**` (matching the spec text verbatim, covering the legacy dir on projects that haven't migrated) and an equivalent rule (or reliance on the scanner's pre-existing hardcoded self-exclusion, which already covers `.cc-os/**`) — so that on a project already migrated to `.cc-os/dochygiene/`, the IGNORE surface still holds. This design does not change the scanner's existing hardcoded self-exclusion; the rulebook's IGNORE rule list is additive to it, not a replacement, so this is satisfied without a code change beyond confirming the shipped `rulebook.json`'s IGNORE entries include `.dochygiene/**` and documenting that `.cc-os/**` is already covered by the pre-existing self-exclusion. ## Risks / Trade-offs - **Two overlapping walk-pruning mechanisms** (the scanner's pre-existing hardcoded `excluded_dirs` and the new rulebook-driven directory-rule prune) risk confusing future maintainers about which one is authoritative for a given path. Mitigated by treating the hardcoded excludes as a distinct, lower-level safety net (self-exclusion of the tool's own state and pre-existing fixture excludes) and the rulebook as the project-configurable lifecycle layer; `scripts/CONTEXT.md` should document both once implemented. - **`git ls-files` / dirty check runs twice** (once at report-build time, once at apply time) for lifecycle deletes — a deliberate, spec-mandated redundancy (ADR-0039: "never trusting the rule … at runtime") rather than an oversight; the cost is one extra git subprocess call per lifecycle entry per run, acceptable given the destructive nature of the operation. - **`extract-then-delete` spans two subsystems** (a generative rewrite via the existing Sonnet-distillation path, then a `git rm`) inside a single op type, and for cross-repo extraction additionally calls into `/os-vault:write`, a different plugin's skill. This is more moving parts in one applied entry than any existing op; a partial failure (extraction succeeds, delete fails, or vice versa) needs care to keep the "one hygiene commit" invariant intact — this design treats extraction-then-delete as atomic per entry (both steps land in the same commit, or neither does), which may need revisiting once implemented against real vault-write latency/failure modes. - **Directory-rule aggregate entries change the meaning of "one entry per file"** that today's `doc-check`/`report-schema` specs assume throughout (e.g. "every `entries[].path` SHALL be a member of `shortlist`" — a directory aggregate entry's path is a directory, not a file, which is a new shape for `path` that existing consumers may not expect). The delta specs below extend rather than replace this requirement, but any code outside this change's touched files that assumes `path` is always a regular file should be audited during implementation. - **`rulebook.py`'s hard requirement on Python ≥3.13** (`glob.translate`) is new relative to the rest of the scripts directory (whose stdlib-only policy has not previously pinned a Python floor this high); if the environment running `os-doc-hygiene` scripts is older, the whole rulebook layer hard-fails to import. This is accepted as a locked design decision (spec §2), not reopened here, but is called out as an operational risk worth a version check with a clear error message at load time. ## Ambiguities resolved while writing this design (Reported per instructions — none silently decided against the spec; these are places the locked documents left an implementation-level gap.) 1. **IGNORE-surface entries vs. ordinary directory-rule "prune + aggregate entry" behavior.** Spec §1/§2 phrasing could be read as saying the IGNORE surface *itself* produces an aggregate shortlist entry ("a directory-rule match prunes the walk and emits one aggregate entry — this IS the ignore surface"), but spec §1 and the §3 tier table also say IGNORE-surface members are simply "never walked," with no entry language. Resolved by treating "prune + aggregate entry" as the general mechanism for lifecycle directory rules with a real lifetime (temporary / delete-once-served), and reserving true zero-signal, zero-entry pruning for the specific IGNORE seed list (`graphify-out/**`, `.dochygiene/**`). Flagging this because it affects whether IGNORE members ever appear anywhere in a report — this design says they never do. 2. **`conventions.json` file shape.** The spec names the file and its v1 contents (archive-bucket, status-frontmatter) and per-entry conceptual fields (name, what it proves, template, pitch) but does not give a JSON schema. This design treats it as a flat array of objects with those four conceptual fields, no envelope/schema_version wrapper (unlike `rulebook.json`, which the spec explicitly gives an envelope). Flagging because a schema_version envelope could be added instead if implementation prefers consistency with rulebook.json; nothing in the spec forecloses either choice. 3. **Where the `git ls-files`/dirty runtime check physically lives** (spec says "verified at runtime via git ls-files + dirty check, never trusting the rule" but doesn't say whether that's a report-build-time check, an apply-time check, or both). This design does both — advisory at report-build time (so the report's tier reflects current reality when shown to the human), authoritative at apply time (so a stale report never causes an unsafe auto-delete) — matching the existing content-hash-guard pattern (`expected_sha256` is likewise computed at build time and re-verified at apply time). Flagging because the spec's single sentence doesn't explicitly mandate the double-check; it was inferred from ADR-0039's "never trusting the rule" plus the existing applier's re-verification precedent (`patch_applier.py`'s content-hash guard). 4. **Extraction-failure handling within `extract-then-delete`** (does a failed extraction hard-fail the whole clean run, or just skip that entry?) is not addressed in the spec. This design treats it as a per-entry skip (consistent with `doc-clean`'s existing partial-success semantics), not a run-level hard failure, unless the failure mode matches one of the existing hard-failure triggers (applier exit 2, write error). Flagging since this determines commit granularity under failure.