cc-os/plugins/os-doc-hygiene/examples/golden/classifier/README.md

Classifier golden examples

Layer-2 reversion-protection fixtures for the classifier (design D7, openspec/changes/add-check/design.md). Each case is a small static fixture doc-tree (input/, stable bytes → stable sha256) paired with expected.json, a full schema-valid machine report representing the expected classification of that tree. These are distinct from the schema-shape fixtures one level up (../valid_report.json / ../invalid_report.json), which only exercise validate_report.py.

Adding or changing a classifier golden is human-gated per the META-RULE in invariants.md.

Cases

Dir	class / subtype	op_type	exact_edit.kind	tier	scanner signal
1-orphaned	stale / orphaned	deterministic	delete-range	confirm	broken_reference
2-superseded	stale / superseded	deterministic	move-to-archive	auto	stale_name_location
3-completed-in-place	stale / completed-in-place	deterministic	insert-frontmatter	auto	archive_to_live_ratio
4-duplicated	stale / duplicated	deterministic	dedupe	auto	stale_name_location (see note)
5-distill	bloat / distill	generative	(none)	confirm	archive_to_live_ratio

Each case directory holds:

• input/ — the fixture doc-tree (the bytes the scanner and the model see).
• proposals.json — the hand-authored model-judgment proposal(s) used to generate expected.json (kept so the computed fields are reproducible).
• expected.json — the full schema-valid machine report (the frozen expectation). Computed fields (expected_sha256, raw_tokens, derived safety_tier) are real: generated by piping a real scanner.py artifact + proposals.json through report_builder.py. scan.project_root is normalized to <input> for portability; generated_at timestamps are wall-clock and intentionally not asserted on.
• notes.md — the one taxonomy edge + scanner signal the case pins.

Note (case 4): the scanner has no dedicated duplication signal. Case 4 rides a co-occurring stale_name_location ("legacy") as its deterministic trigger; the duplicated judgment itself is the model's. See 4-duplicated/notes.md.

Two harnesses

Hermetic unit harness (in pytest — tests/test_classifier_golden.py)

Runs with NO network and NO model. For each case it asserts only the deterministic / stable parts: scanner signals on the expected paths, validate_report.py accepts each expected.json, and internal consistency of the stable classification fields (closed enums, op_type, exact_edit.kind, the derived safety_tier recomputed, and expected_sha256 re-verified against the fixture bytes). This is the regression gate that runs in CI.

Live model-classification regression (OUTSIDE pytest — manual / agent-invoked)

This is the model-dependent check; it is intentionally NOT part of the unit suite because it calls a live model. Run it manually (or have an agent run it) when you want to confirm the classifier still reaches the expected judgment.

For each case directory:

1. Run an actual classification over input/ — either /os-doc-hygiene:check --scope <case>/input or by invoking the check skill against that tree. This produces a fresh machine report.

   Confirm the invocation once the check command lands. --scope is a filter on a scan rooted at the resolved project root, not a re-root — to classify a case in isolation you want the check rooted at <case>/input (as the hermetic suite roots the scanner). Verify the exact flag/rooting against the shipped skills/check/SKILL.md before relying on it.

2. Diff the produced classification's stable fields against expected.json, per entry:

   • category.class
   • category.subtype
   • op_type
   • the derived safety_tier
   • exact_edit.kind

   A mismatch on any of these is a regression — investigate before accepting.

3. Advisory only (flag for human review, do not auto-fail):

   • op prose wording.
   • exact anchor line numbers (exact_edit.anchor.start_line / end_line) and reducible_range.

   These are model-authored and may drift slightly without indicating a real classification regression; surface them for a human to eyeball rather than gating on them.

Because expected.json encodes a captured model judgment, any intended change to the expected classification is a golden change → human-gated per the META-RULE.

Regenerating an expected.json (after an approved fixture change)

python3 scripts/scanner.py --root examples/golden/classifier/<case>/input > /tmp/scan.json
python3 scripts/report_builder.py \
    --scan /tmp/scan.json \
    --proposals examples/golden/classifier/<case>/proposals.json \
    --root examples/golden/classifier/<case>/input \
    --out-json examples/golden/classifier/<case>/expected.json
# then normalize scan.project_root to "<input>" and re-run the hermetic suite.

raw_tokens reflect whichever token-estimator backend was active at generation time (the zero-dependency heuristic here, since no tiktoken vocab is vendored). The hermetic suite does not assert on raw_tokens, so regenerating on a tiktoken-equipped machine may shift those counts harmlessly — don't mistake that diff for a regression.