jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-doc-hygiene/skills/calibrate/SKILL.md

25 KiB
Raw Blame History

description
Learn new lifecycle rules for a project by clustering unmatched files, nominating candidate globs (cheap model), and having a strong model judge and confirm/reject/amend them, with a mandatory rule report to the human before any persistence. Invoked by `/os-doc-hygiene:calibrate`.

Hygiene Calibrate Skill

Orchestrates the learn-new-rules loop (lifecycle-spec.md §8): cluster-and- sample → cheap-model nominate → deterministic intake filter (drop repeat rejections, carry related rejections + open consults forward) → strong- model judge → rule report (human, including open consults) → persist → retest. It runs over the unmatched pool (unmatched = unmanaged = not governed by any existing rulebook rule, per rulebook.py), and is the only new skill this change adds — check/clean are unchanged in structure (ADR-0039/-0041, lifecycle-spec.md §7).

All scripts live under ${CLAUDE_PLUGIN_ROOT}/scripts/. Run them with python3 from the user's project directory (cwd). Use the session scratchpad directory for all intermediate artifacts.

Precondition: requires CLAUDE_PLUGIN_ROOT. Every script path resolves against it; abort rather than guessing a path if it is unset.

Pick a scratch dir once and reuse it: SCRATCH="$(mktemp -d)".

What this skill NEVER does

  • It never applies a rule without the human having seen the Step 4 rule report first (spec: "no rule shall be persisted before this report has been shown").
  • It never removes a rule automatically — removals are HITL-only in all cases, with recorded reasoning (spec §5).
  • It never writes to the global plugins/os-doc-hygiene/rulebook.json without an explicit, distinct confirmation beyond project-rule confirmation (cross-repo write into cc-os).
  • It never applies a drafted convention adoption (§6 below) without explicit human confirmation.

Workflow

Step 1 — (D) Load the rulebook and scan for the unmatched pool

Same rulebook-load pattern as check's Step 0.5, but here the candidate pool is the unmatched paths — files the current rulebook leaves ungoverned — not the signal-bearing shortlist.

export SCRATCH
python3 -c '
import json, os, sys
from pathlib import Path
sys.path.insert(0, os.environ["CLAUDE_PLUGIN_ROOT"] + "/scripts")
from rulebook import load_rulebook, RulebookLoadError
from scanner import Scanner, _resolve_project_root, _git_log_real, _git_commit_time_real

root = _resolve_project_root(Path.cwd())
project_rules = root / ".dochygiene-rules.json"
try:
    rulebook = load_rulebook(project_path=project_rules if project_rules.is_file() else None)
except RulebookLoadError as exc:
    print(json.dumps({"error": "rulebook-load-failed", "detail": str(exc)}))
    sys.exit(2)

scanner = Scanner(
    root=root, rulebook=rulebook,
    git_log_fn=_git_log_real, git_commit_time_fn=_git_commit_time_real,
)
artifact = scanner.run()

# The unmatched pool is every file the scan encountered that carries no
# lifecycle signal AND is not itself an IGNORE-pruned/directory-rule
# aggregate entry -- i.e. shortlist entries with no rulebook governance.
signals = artifact.get("signals", {})
unmatched = [p for p in artifact.get("shortlist", []) if p not in signals]

Path(os.environ["SCRATCH"] + "/scan.json").write_text(json.dumps(artifact, indent=2))
Path(os.environ["SCRATCH"] + "/unmatched.json").write_text(json.dumps(unmatched, indent=2))
print(f"unmatched pool: {len(unmatched)} paths")
'
  • Exit 2 / rulebook load failure → hard STOP, same as check Step 0.5. Do not proceed with a silently-empty rulebook.
  • Note: signals here means ANY signal (stale/bloat/lifecycle) — a file with a stale/bloat signal but no lifecycle rule match is still "unmatched" with respect to the rulebook, and belongs in the pool. Filter precisely on lifecycle-named signals if the project has files carrying only non-lifecycle signals that should stay in the pool:
unmatched = [
    p for p in artifact["shortlist"]
    if not any(s.get("name") == "lifecycle" for s in artifact.get("signals", {}).get(p, []))
]

Use this refined filter, not the simpler one above, when signals may carry non-lifecycle entries for shortlisted paths.

If unmatched is empty → report "Nothing to calibrate — every shortlisted file is already governed by a rulebook rule." STOP.

Step 2 — (D) Cluster and sample — calibrate_helpers.ClusterSampler

Deterministic, no model. Groups unmatched paths by path-shape (directory prefix + filename shape class — digit runs collapse to #, hex-looking runs collapse to ~) and samples representatives per cluster, capped.

python3 -c '
import json, os, sys
from pathlib import Path
sys.path.insert(0, os.environ["CLAUDE_PLUGIN_ROOT"] + "/scripts")
from calibrate_helpers import ClusterSampler

unmatched = json.loads(Path(os.environ["SCRATCH"] + "/unmatched.json").read_text())
clusters = ClusterSampler().cluster_to_dicts(unmatched)
Path(os.environ["SCRATCH"] + "/clusters.json").write_text(json.dumps(clusters, indent=2))
print(f"{len(clusters)} clusters")
'

Each cluster is {key, dir_prefix, shape, total, sample}. Rules are always proposed against a cluster, never a single instance in isolation (spec: "the nomination is derived from a cluster of similar unmatched paths").

Step 3 — (M) Cheap-model nomination — haiku subagent, one per cluster

For each cluster, dispatch a haiku subagent (LOOP-GUARD: point it at workflows/nominate.md, never this SKILL.md) constrained to nominate a bare glob pattern + candidate lifetime — patterns only, never an exact-instance glob (a run-id, hash, or bare timestamp hardcoded into the glob).

Agent tool parameters:
- subagent_type: "general-purpose"
- model: haiku
- description: "Nominate lifecycle rule for cluster: <cluster.key>"
- prompt: |
    Read and follow the workflow at:
    ${CLAUDE_PLUGIN_ROOT}/skills/calibrate/workflows/nominate.md

    Cluster: <cluster.dir_prefix> / shape <cluster.shape>
    Total matching paths: <cluster.total>
    Sample paths:
    <cluster.sample, one per line>

    Return ONLY the JSON object specified in the workflow.

Collect all nominations into "$SCRATCH/nominations.json" (array, one per cluster, tagged with the originating cluster.key).

Do not trust a haiku nomination as final. The "class, never path" rule-quality test is enforced by the strong-model judge (Step 4) plus the deterministic RuleQualityChecker (Step 5's report), never accepted from haiku at face value.

Step 3.5 — (D) Nomination intake filter — calibrate_helpers.NominationIntakeFilter

Deterministic, no model (lifecycle-spec.md §8 step 3; NominationIntakeFilter requirement). Reads the project rules file's nominations key via RulesFileWriter.load, then drops any nomination that exactly repeats a rejected glob+lifetime, annotates survivors with every related rejection (match-set intersection on the current shortlist), and passes ALL open consults through unconditionally — this is the input the judge prompt's "Nominations memory" section (Step 4) consumes.

python3 -c '
import json, os, sys
from pathlib import Path
sys.path.insert(0, os.environ["CLAUDE_PLUGIN_ROOT"] + "/scripts")
from calibrate_helpers import RulesFileWriter, NominationIntakeFilter

scan = json.loads(Path(os.environ["SCRATCH"] + "/scan.json").read_text())
project_rules = Path(scan["project_root"]) / ".dochygiene-rules.json"
nominations = json.loads(Path(os.environ["SCRATCH"] + "/nominations.json").read_text())
shortlist = scan.get("shortlist", [])

writer = RulesFileWriter()
data, load_warnings = writer.load(project_rules)
memory = data.get("nominations", {})

result = NominationIntakeFilter(
    rejected=memory.get("rejected", []),
    consults=memory.get("consults", []),
).filter(nominations, shortlist)

Path(os.environ["SCRATCH"] + "/intake.json").write_text(json.dumps(result, indent=2))
print(f"{len(result[\"survivors\"])} survivors, {len(result[\"dropped\"])} dropped, {len(result[\"consults\"])} open consults")
if result["dropped"]:
    for d in result["dropped"]:
        print(f"  dropped: {d[\"glob\"]} -> {d[\"lifetime\"]} ({d[\"reason\"]})")
'
  • Surface every drop in the run summary shown to the human alongside the Step 5 report — a dropped nomination never reaches the judge, so this is the only place it is visible.
  • intake.json's survivors (each nomination plus its related_rejections annotation) and consults (all open consults, unconditionally) are what gets embedded in Step 4's judge prompt as the "Nominations memory" input section — feed the whole survivors array (not the raw nominations.json) forward into Step 4, and include consults even when empty (an empty array is a valid, meaningful "no open consults" signal).

Step 4 — (M) Strong-model batched judgment — ONE Opus/Fable subagent

Dispatch a single batched strong-model subagent (model: opus, or the project's configured Fable-tier model) to judge ALL nominations from Step 3.5 in one call (LOOP-GUARD: point it at workflows/judge.md, never this SKILL.md). The judge gathers its OWN evidence — re-reads matched paths against the live tree, checks near-miss boundaries — rather than trusting the haiku nomination's claims.

Judge on intake.json's survivors (each nomination annotated with its related_rejections), never the raw nominations.json — the dropped exact-repeats never reach this step. intake.json's consults is the "Open consults" input regardless of what haiku nominated this round.

Agent tool parameters:
- subagent_type: "general-purpose"
- model: opus
- description: "Judge doc-hygiene calibration nominations"
- prompt: |
    Read and follow the workflow at:
    ${CLAUDE_PLUGIN_ROOT}/skills/calibrate/workflows/judge.md

    Project root: <scan.project_root>

    Nominations to judge (verbatim, each with related_rejections):
    <contents of $SCRATCH/intake.json's "survivors">

    Nominations memory — open consults (unconditional, may be empty):
    <contents of $SCRATCH/intake.json's "consults">

    Seed intake: <see "Seed intake" below — include or omit per pass>

    Return ONLY the JSON array of verdicts specified in the workflow.

Verdicts are exactly one of confirm / reject / amend / consult. consult is MANDATORY whenever the judge cannot determine if an artifact is regenerable or must be retained — never resolved to confirm or reject in that case. Write the judge's verdict array to "$SCRATCH/verdicts.json".

Seed intake: the #41 clutter-inventory seed candidates enter at THIS step (judge intake), for every calibration run except cc-os calibration pass #1, which withholds them as a sealed answer key (one-off carve-out, see lifecycle-spec.md §9). If this run IS cc-os pass #1, do NOT include seed candidates in the judge prompt. Every other run (including later cc-os runs) includes full seed intake.

Step 5 — (D) Rule report to the human — BEFORE any persistence

Deterministic, no model — calibrate_helpers.RuleReportBuilder plus RuleQualityChecker. For every judge verdict of confirm or amend (never for reject/consult — those are not proposed for persistence), assemble the 5-element report and run the quality lints:

python3 -c '
import json, os, sys
from pathlib import Path
sys.path.insert(0, os.environ["CLAUDE_PLUGIN_ROOT"] + "/scripts")
from calibrate_helpers import RuleReportBuilder, RuleQualityChecker

scan = json.loads(Path(os.environ["SCRATCH"] + "/scan.json").read_text())
verdicts = json.loads(Path(os.environ["SCRATCH"] + "/verdicts.json").read_text())
all_paths = scan.get("shortlist", [])

proposed = [v["rule"] for v in verdicts if v["verdict"] in ("confirm", "amend")]
builder = RuleReportBuilder()
checker = RuleQualityChecker()

report = []
for rule in proposed:
    entry = builder.build(rule, all_paths).to_dict()
    entry["quality"] = {
        "class_not_path": checker.class_not_path(rule["glob"], all_paths).to_dict(),
    }
    report.append(entry)

Path(os.environ["SCRATCH"] + "/rule_report.json").write_text(json.dumps(report, indent=2))
print(json.dumps(report, indent=2))
'

Render this to the human as patterns and examples, not JSON schema — per rule:

Proposed rule: <glob verbatim>
  Lifetime: <lifetime>   Tier: <auto|confirm>
  Matches (<total>): <sample paths, one per line> [+ N more]
  Near-miss (does NOT match, but looks similar): <near-miss paths, or "(none)">
  Why: <plain-language "what this is and why it's clutter">
  Quality check: <PASS, or the flagged reason if class_not_path failed>

If class_not_path failed (a glob that can, by construction, only ever match one file — a failed generalization), flag it LOUDLY in the rendered report rather than silently dropping or persisting it; ask the human whether to have the judge re-amend it (loop back to Step 4 for that one rule) or drop it from this round.

Open consults section

List every entry from $SCRATCH/intake.json's consults — the open consults from prior runs (plus any new consult verdict this run) — one per entry:

Open consult: <glob>
  Asked: <asked_on>   Cluster: <cluster_key>
  Question: <question>
  Evidence: <evidence>

Each open consult has exactly three exits, all human, none of which is gated the way a rule confirmation is:

  • (a) Answer settles the purpose — the human's answer determines a lifetime; persist a normal rule through this same report/Step 6 flow AND delete the consult entry, in the same write (the rule supersedes it).
  • (b) Not rule-worthy — the human decides the artifact class isn't a rule; rewrite the entry into nominations.rejected with rejected_by: "human" and the human's stated why.
  • (c) Defer — no answer this round; the entry stays as-is and resurfaces in the next :calibrate run's Step 3.5/4/5.

New rejections and consults produced this run (from judge consult verdicts, or from exit (b) above) appear in this report for visibility but are not individually gated — unlike proposed rules, they carry no deletion authority, only memory. Only proposed rules go through the "Persist these N project rules?" gate below.

No rule is written anywhere until the human has seen this report and responded. Ask: "Persist these N project rules? (yes / no / a subset by number)". Separately, resolve each open consult per its three exits above. Only proceed to Step 6 for the rules the human approves and the consults the human has settled or declined this round.

Step 6 — (D) Persistence — canonical writer only

Every settled verdict persists through calibrate_helpers.RulesFileWriter (load → mutate the parsed dict → write) — no code path serializes .dochygiene-rules.json by hand. Before running this, translate the human's Step 5 responses into four in-memory lists (the orchestrating skill's own bookkeeping — there is no scratch file for these, since they come from the live conversation, not a subagent):

  • approved_rules — the judge confirm/amend rule objects the human said yes to (including keep-lifetime rules and Step-5-exit-(a) rules that answer an open consult).
  • declined — (rule, why) pairs for judge-proposed rules the human said no to at the report.
  • settled_consult_globs — globs of open consults the human resolved this run, either via exit (a) (answered, folded into approved_rules above) or exit (b) (declined, folded into new_human_rejections below); these are removed from nominations.consults in the same write.
  • new_human_rejections — (glob, lifetime, why) for exit-(b) consults ("not rule-worthy"), written with rejected_by: "human".
  • still_open_consults — judge consult verdicts from THIS run ({glob, question, evidence, cluster_key}) that remain unresolved after Step 5 (exit (c), or simply not reached this round).
python3 -c '
import json, os, sys
from datetime import date
from pathlib import Path
sys.path.insert(0, os.environ["CLAUDE_PLUGIN_ROOT"] + "/scripts")
from calibrate_helpers import RulesFileWriter

scan = json.loads(Path(os.environ["SCRATCH"] + "/scan.json").read_text())
project_rules = Path(scan["project_root"]) / ".dochygiene-rules.json"
today = date.today().isoformat()

# approved_rules / declined / settled_consult_globs / new_human_rejections /
# still_open_consults come from the human'"'"'s Step 5 responses (see above).

writer = RulesFileWriter()
data, load_warnings = writer.load(project_rules)
nominations = data.setdefault("nominations", {"consults": [], "rejected": []})

# 1. Judge confirm/amend verdicts the human approved -> plain rules
#    (keep verdicts and exact-path keep singletons allowed -- the keep-tier
#    relaxation). confirmed_by/confirmed_on are set here, never by the judge.
for rule in approved_rules:
    rule["confirmed_by"] = "human"
    rule["confirmed_on"] = today
    data["rules"].append(rule)

# 2. Human declines at the rule report -> rejected entries, so a later
#    haiku round cannot re-nominate the identical glob+lifetime without the
#    judge knowing.
for rule, why in declined:
    nominations["rejected"].append({
        "glob": rule["glob"], "lifetime": rule["lifetime"],
        "why": why, "rejected_by": "human", "judged_on": today,
    })

# 3. Consults the human declined this run (Step 5 exit b) -> rejected
#    entries too, same rejected_by/judged_on convention.
for glob_pattern, lifetime, why in new_human_rejections:
    nominations["rejected"].append({
        "glob": glob_pattern, "lifetime": lifetime,
        "why": why, "rejected_by": "human", "judged_on": today,
    })

# 4. Still-open consults from this run -> nominations.consults, deduped by
#    glob (an existing entry with the same glob wins -- never duplicated).
existing_consult_globs = {c["glob"] for c in nominations["consults"]}
for consult in still_open_consults:
    if consult["glob"] not in existing_consult_globs:
        nominations["consults"].append({**consult, "asked_on": today})

# 5. Consults the human just settled this run (answered or declined) leave
#    nominations.consults -- the rule (step 1) or rejection (step 3)
#    supersedes them.
nominations["consults"] = [
    c for c in nominations["consults"] if c["glob"] not in settled_consult_globs
]

write_warnings = writer.write(project_rules, data)
'
  • Project rules (the common case): land in <project-root>/.dochygiene- rules.json on judge confirm/amend PLUS this step's human approval — including plain lifetime: keep rules for judge keep-purpose verdicts, exact-path singletons allowed under the keep-tier relaxation.
  • Human declines at the rule report persist as rejected entries with rejected_by: "human" (never silently dropped) — see item 2 above.
  • Open consult verdicts persist to nominations.consults, deduped by glob at write time — see item 4 above. An answered consult (Step 5 exit a) is deleted from consults in the same write that persists its superseding rule; a declined consult (exit b) is deleted from consults in the same write that adds its rejected entry; a deferred consult (exit c) is left untouched and resurfaces next run.
  • Global rulebook writes (plugins/os-doc-hygiene/rulebook.json) require a SEPARATE, EXPLICIT confirmation beyond the project-rule approval above — this is a cross-repo write into cc-os itself. Ask distinctly: "This rule looks like it belongs in the GLOBAL rulebook (applies to every project), not just this one. Write it to the global rulebook instead/as well? (yes/ no)". Only write on explicit "yes" to THIS question. (Rejections/consults are always project-scoped memory — never written to the global rulebook.)
  • Removals are HITL-only, always, regardless of scope: only remove a rule when the human explicitly asks to, with the reasoning recorded in the rule's own note field (or a comment in the calibration run's summary) — never as an automatic side effect of a calibration pass. New rejections and consults are memory, not deletion authority, and are never gated the way a rule persist/remove is (Step 5).

Each persisted rule gets confirmed_by (the human's decision, not the judge's — a model-proposed rule may never set confirm: true on itself, it may only ask) and confirmed_on (today's date) per the rulebook schema (rulebook.py's _KNOWN_FIELDS).

Consult loop — worked example (map #49, #56/#59)

Worked example of consult persistence (lifecycle-spec.md §2 "Nominations memory"), wired through Steps 3.5/5/6 above. Run 1: the judge returns consult on docs/orchestration-audit/*.md ("retained audit trail, or disposable once the tune-up lands?"). Step 6's writer persists it to nominations.consults (glob, question, evidence, cluster_key, asked_on — deliberately no lifetime). Run 2, weeks later: the deterministic intake filter injects the still-open consult into the judge prompt's "Nominations memory" section, AND the Step 5 report renders it under "Open consults". Three exits, all human: (a) the human answers "audit trail" → a lifetime: keep rule is persisted and the consult entry deleted (the rule supersedes it); (b) "not rule-worthy" → the entry is rewritten into nominations.rejected with the human's why (rejected_by: "human"); (c) no answer → the entry stays and resurfaces on run 3. A consult never filters files and never expires on its own.

Step 7 — (D) Retest loop

Re-run Steps 1-6 (including 3.5) against the shrunk unmatched pool. Stop when:

  • a round yields fewer than 2 new persisted rules, OR
  • the unmatched pool shrank by less than 10% since the previous round,
  • hard cap: 3 rounds, regardless of shrink rate.

Track round count and the unmatched-pool size at the start of each round in the scratch dir ($SCRATCH/round_N_unmatched_count.txt) to compute shrink %.

§6 (design.md) — draft convention adoption, never apply unasked

While reviewing the unmatched pool, :calibrate MAY notice a pattern that would benefit from a conventions.json convention (archive-bucket or status-frontmatter) rather than a plain rule. If so, it MAY draft the adoption — the graduated rule (e.g. served_when_path: <dir>/archive/{name}) PLUS the concrete file moves or frontmatter additions the convention implies — and present it to the human alongside the Step 5 rule report, for approval. It never applies a drafted adoption without explicit confirmation — no rulebook write and no file move happens until the human confirms.

Calibration pass #1 (cc-os) — special-case reminders

See lifecycle-spec.md §9 and openspec task group 6 for the full protocol. When run is explicitly cc-os pass #1:

  • Withhold the #41 seed candidates from judge intake (Step 4) — do NOT paste them into the judge prompt.
  • The protected set is a hard gate: if ANY rule proposed for persistence (Step 5/6) has a glob matching a path in the protected set (eval scenarios//scenarios-reserve//fixture//judge-rubric.md; openspec/specs/; docs/adr/**; mirrored .claude//.codex//.pi/ skill dirs; CLAUDE.md; plugin source), REFUSE to persist that rule and flag it loudly — regardless of the tier the judge assigned it. A consult verdict touching a protected path during exploration is free (does not fail the pass) as long as it is never persisted.
  • Do not treat this carve-out as a permanent behavior — every later run (including future cc-os runs) uses full seed intake.

Invariants

  • Steps 1, 2, 3.5, 5, 6 are deterministic scripts/logic — no model.
  • Step 3 = haiku (cheap, per-cluster nomination, patterns only).
  • Step 3.5 = the deterministic NominationIntakeFilter — exact glob+lifetime repeats of a rejected entry are dropped before the judge ever sees them; survivors carry related_rejections; all open consults pass through unconditionally.
  • Step 4 = ONE batched Opus/Fable judge call, never per-cluster.
  • No rule is persisted before the Step 5 report has been shown to the human (hard invariant — never skip Step 5, never merge it with Step 6).
  • Global-rulebook writes require a SEPARATE explicit confirmation beyond project-rule approval.
  • Removals are HITL-only in all cases, with recorded reasoning.
  • A model-proposed rule may never self-set confirm: true — only the human's Step 5/6 response does.
  • Retest loop stops at <2 new rules OR <10% shrink; hard cap 3 rounds.
  • LOOP GUARD: the nominate subagent prompt MUST point to workflows/nominate.md; the judge subagent prompt MUST point to workflows/judge.md. Neither ever points to this SKILL.md.
  • SUBAGENT AUTHORIZATION: both subagents are executors — authorization is terminal. Neither re-asks for approval; if either objects, REPORT-AND-EXIT and let the orchestrator (this skill, or ultimately the human at Step 5/6) adjudicate.