108 lines
5.3 KiB
Markdown
108 lines
5.3 KiB
Markdown
# Classifier golden examples
|
|
|
|
Layer-2 reversion-protection fixtures for the **classifier** (design D7,
|
|
`openspec/changes/add-check/design.md`). Each case is a small **static** fixture
|
|
doc-tree (`input/`, stable bytes → stable sha256) paired with `expected.json`, a
|
|
full schema-valid machine report representing the expected classification of that
|
|
tree. These are **distinct** from the schema-shape fixtures one level up
|
|
(`../valid_report.json` / `../invalid_report.json`), which only exercise
|
|
`validate_report.py`.
|
|
|
|
Adding or changing a classifier golden is **human-gated** per the META-RULE in
|
|
`invariants.md`.
|
|
|
|
## Cases
|
|
|
|
| Dir | class / subtype | op_type | exact_edit.kind | tier | scanner signal |
|
|
|-----|-----------------|---------|-----------------|------|----------------|
|
|
| `1-orphaned` | stale / orphaned | deterministic | `delete-range` | confirm | `broken_reference` |
|
|
| `2-superseded` | stale / superseded | deterministic | `move-to-archive` | auto | `stale_name_location` |
|
|
| `3-completed-in-place` | stale / completed-in-place | deterministic | `insert-frontmatter` | auto | `archive_to_live_ratio` |
|
|
| `4-duplicated` | stale / duplicated | deterministic | `dedupe` | auto | `stale_name_location` (see note) |
|
|
| `5-distill` | bloat / distill | generative | (none) | confirm | `archive_to_live_ratio` |
|
|
|
|
Each case directory holds:
|
|
|
|
- `input/` — the fixture doc-tree (the bytes the scanner and the model see).
|
|
- `proposals.json` — the hand-authored model-judgment proposal(s) used to
|
|
*generate* `expected.json` (kept so the computed fields are reproducible).
|
|
- `expected.json` — the full schema-valid machine report (the frozen expectation).
|
|
Computed fields (`expected_sha256`, `raw_tokens`, derived `safety_tier`) are
|
|
**real**: generated by piping a real `scanner.py` artifact + `proposals.json`
|
|
through `report_builder.py`. `scan.project_root` is normalized to `<input>` for
|
|
portability; `generated_at` timestamps are wall-clock and intentionally not
|
|
asserted on.
|
|
- `notes.md` — the one taxonomy edge + scanner signal the case pins.
|
|
|
|
> **Note (case 4):** the scanner has no dedicated duplication signal. Case 4 rides
|
|
> a co-occurring `stale_name_location` ("legacy") as its deterministic trigger; the
|
|
> *duplicated* judgment itself is the model's. See `4-duplicated/notes.md`.
|
|
|
|
## Two harnesses
|
|
|
|
### Hermetic unit harness (in pytest — `tests/test_classifier_golden.py`)
|
|
|
|
Runs with **NO network and NO model**. For each case it asserts only the
|
|
deterministic / stable parts: scanner signals on the expected paths,
|
|
`validate_report.py` accepts each `expected.json`, and internal consistency of the
|
|
stable classification fields (closed enums, op_type, `exact_edit.kind`, the derived
|
|
`safety_tier` recomputed, and `expected_sha256` re-verified against the fixture
|
|
bytes). This is the regression gate that runs in CI.
|
|
|
|
### Live model-classification regression (OUTSIDE pytest — manual / agent-invoked)
|
|
|
|
This is the **model-dependent** check; it is intentionally NOT part of the unit
|
|
suite because it calls a live model. Run it manually (or have an agent run it) when
|
|
you want to confirm the classifier still reaches the expected judgment.
|
|
|
|
For each case directory:
|
|
|
|
1. Run an actual classification over `input/` — either `/hygiene check
|
|
--scope <case>/input` or by invoking the `hygiene-check` skill against that
|
|
tree. This produces a fresh machine report.
|
|
|
|
> **Confirm the invocation once the check command lands.** `--scope` is a
|
|
> filter on a scan rooted at the resolved project root, not a re-root — to
|
|
> classify a case in isolation you want the check *rooted at*
|
|
> `<case>/input` (as the hermetic suite roots the scanner). Verify the exact
|
|
> flag/rooting against the shipped `commands/hygiene.md` before relying on it.
|
|
2. Diff the produced classification's **stable** fields against `expected.json`,
|
|
per entry:
|
|
- `category.class`
|
|
- `category.subtype`
|
|
- `op_type`
|
|
- the derived `safety_tier`
|
|
- `exact_edit.kind`
|
|
|
|
A mismatch on any of these is a **regression** — investigate before accepting.
|
|
|
|
3. **Advisory only** (flag for human review, do **not** auto-fail):
|
|
- `op` prose wording.
|
|
- exact anchor line numbers (`exact_edit.anchor.start_line` / `end_line`) and
|
|
`reducible_range`.
|
|
|
|
These are model-authored and may drift slightly without indicating a real
|
|
classification regression; surface them for a human to eyeball rather than
|
|
gating on them.
|
|
|
|
Because `expected.json` encodes a captured model judgment, any *intended* change to
|
|
the expected classification is a golden change → **human-gated** per the META-RULE.
|
|
|
|
## Regenerating an `expected.json` (after an approved fixture change)
|
|
|
|
```
|
|
python3 scripts/scanner.py --root examples/golden/classifier/<case>/input > /tmp/scan.json
|
|
python3 scripts/report_builder.py \
|
|
--scan /tmp/scan.json \
|
|
--proposals examples/golden/classifier/<case>/proposals.json \
|
|
--root examples/golden/classifier/<case>/input \
|
|
--out-json examples/golden/classifier/<case>/expected.json
|
|
# then normalize scan.project_root to "<input>" and re-run the hermetic suite.
|
|
```
|
|
|
|
`raw_tokens` reflect whichever token-estimator backend was active at generation
|
|
time (the zero-dependency heuristic here, since no tiktoken vocab is vendored).
|
|
The hermetic suite does **not** assert on `raw_tokens`, so regenerating on a
|
|
tiktoken-equipped machine may shift those counts harmlessly — don't mistake that
|
|
diff for a regression.
|