# os-adr Eval C — ambiguity-ladder discrimination eval (held-out) _Last updated: 2026-07-06 — harness built and self-tested (all scenarios pass model-free checks)._ Measures **generalization across decreasing cue explicitness** — does the model correctly trigger `/os-adr:find` or `/os-adr:create` as architectural decision vocabulary becomes less explicit, and equally important, **does it correctly NOT trigger when no decided constraint is in play**? This is the discrimination challenge: both false positives (over-trigger) and false negatives (miss the decision) are failures. Eval B (unprompted-behavior baseline) optimized its wording against its own grid for 5 iterations (2026-07-04), so its 8/8 (sonnet) / 7/8 (haiku) are training-set scores. Eval C measures whether that learned behavior generalizes to **different project, different target ADRs, and most critically, paired negative scenarios that ensure the model isn't just learning "always trigger"**. > **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and > `scenarios-reserve/*.md` must never be pasted into an interactive session or informally > "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates > transcripts and never uses the Task blocks. Informal trials contaminate the measurement. ## Measurement discipline This eval is a **frozen measurement pass** — it runs once with a baseline grid. The moment anyone iterates wording against the run-set results, this set is contaminated; future wording work **must switch to the reserve-set** for measurement. See "Reserve set protection" below. ## Design: ambiguity ladder with paired positive/negative ### The ladder (3 levels) Each level decreases explicit architectural decision cues: - **Level 1 (Explicit):** File paths named, implementation vocabulary explicit ("timeout", "retry", "backoff", "exponential"), decision nature overt. - **Level 2 (Moderate):** Relevant vocabulary present but no file paths or explicit decision framing; model must recognize architecture-level significance. - **Level 3 (Conceptual):** Intent-framed only (what outcome is needed), zero ADR vocabulary, no file paths. Model must recognize that this is an architectural concern. ### Paired positive/negative at every level For each level, **two scenarios that differ only in whether a decided constraint is in play**: - **Positive (P)**: A decided constraint IS applicable → consulting `/os-adr:find` and/or recording with `/os-adr:create` is correct. - **Negative (N)**: Matched framing (same vocabulary level, same form — question or intent) but NO Accepted ADR constrains the task → consulting/creating an ADR is an over-trigger. The negative scenario is the discrimination test: wording tuned only for sensitivity ("always trigger") will fail negatives. Precision (correct non-triggering) is as important as recall. ### Non-monotonic difficulty expected Difficulty does NOT increase uniformly L1 → L2 → L3. An explicit cue (L1) might be harder to miss than a conceptual one (L3), depending on model tier and wording. Per `running-autoresearch-skill-evals.md`, each level is validated independently and results reported in a per-level grid, not collapsed to a single threshold. ## Two target ADR pairs, independent measurement **Run-set** (6 scenarios, frozen for measurement): - Target ADR 1: 0002 — "Job timeouts after 5 minutes with exponential backoff, max 3 attempts" (Accepted) - Target ADR 2: 0001 → 0002 (Superseded) for distractor validation - Scenarios: P1-L1, N1-L1, P2-L2, N2-L2, P3-L3, N3-L3 (all about job execution timeout/retry) **Reserve-set** (6 scenarios, held until run-set is contaminated by wording iteration): - Target ADR 2: 0003 — "Route all job notifications through central NotificationService" (Accepted) - Scenarios: P4-L1, N4-L1, P5-L2, N5-L2, P6-L3, N6-L3 (all about notification routing) Both sets are valid measurement surfaces; the split is for the protection protocol, not quality difference. ## Scenario framing discipline (critical) The CLAUDE.md instruction carried verbatim in the fixture has two triggers: - Semantic: "architecture-level choice → run `/os-adr:find`" - Mechanical: "before your first edit to any existing source/config file → run `/os-adr:find`" **Negative scenarios avoid the mechanical trigger**: they're framed as pure questions or new module designs, so trigger #2 never fires. Then "no ADR touch" is a clean, valid over-trigger signal. If a negative scenario edits an existing file, the over-trigger criteria shift to "created an ADR claiming a phantom constraint," requiring judge fallback. ## Fixture **Project:** `fixture/project/` — small Python async job queue system (`taskq`). **ADR history:** 6 ADRs (2026-02-01 to 2026-03-01), generated with plugin CLIs: - 0001: "Job timeout at 10s, simple retry" (Superseded) - 0002: "Job timeout at 5 min, exponential backoff, max 3 attempts" (Accepted, supersedes 0001) - 0003: "Route notifications through central service" (Accepted) - 0004: "Persist queue to SQLite" (Accepted) - 0005: "Failed jobs escalate to human review queue" (Accepted) - 0006: "All job state transitions logged to audit trail" (Accepted) Each ADR has `affected-paths` set (e.g., 0002 → [taskq/execution.py, taskq/queue.py]) so `bin/adr-find` can validate path-matching behavior if needed. `.graphifyignore` excludes markdown, so any future graph build (not part of this eval) would be pure-AST / reproducible. ## Two-axis scoring - **Axis (a) — consultation trigger** - **Positive scenarios**: mechanical PASS iff any `tool_use` block invoked an `os-adr:*` skill, `bin/adr-*` CLI, or touched `docs/adr/`. - **Negative scenarios**: INFORMATIONAL ONLY (recorded as `A:yes` or `A:no`, does NOT determine pass/fail). Consultation is acceptable per CLAUDE.md's "cheap, deterministic CLI call" endorsement. The discrimination measurement is not whether the model consulted, but whether it **recorded a false decision** or **asserted a non-existent constraint**. - Hook context (`SessionStart [os-adr]` event) is a precondition: missing hook → FAIL both axes. - **Axis (b) — specific correctness** - **Positive scenarios** (write-trigger): - Mechanical PASS when a new `docs/adr/0007-*.md` exists and its text matches the scenario's topic pattern (regex per scenario class). - Otherwise: judge fallback (haiku, `judge-rubric.md`, reads only final message; stubbed via `ADR_EVAL_C_JUDGE_CMD` for self-tests). - **Negative scenarios** (non-decisional tasks, investigate/test/debug framing): - Mechanical FAIL if a new ADR file was created (over-trigger: creating a decision record for a non-decisional task). This is the ONLY fail condition. - Citation of existing ADRs in the final message is recorded as informational `cited-adr:yes/no`, never a FAIL — the negatives sit deliberately in ADR-covered domains, and truthfully citing a governing ADR while documenting/testing/auditing is correct behavior. Separating true citations from false constraint claims needs semantic judgment; a keyword match fails correct answers. - Otherwise: PASS. **Overall PASS**: - **Positive scenarios**: both axes PASS. - **Negative scenarios**: axis (b) PASS only (axis a is informational, doesn't affect pass/fail). TSV format: `scenario model A:yes|no-or-PASS|FAIL B:PASS|FAIL PASS|FAIL reasons` ## Running the evaluation ### Single cell (manual test) ```bash eval-c/bin/sandbox P2-L2-execution /tmp/adr-eval-c-test eval-c/bin/run P2-L2-execution haiku /tmp/adr-eval-c-test --reps 1 --results /tmp/results.tsv cat /tmp/results.tsv ``` Each rep: fresh sandbox → `claude -p --model --output-format stream-json --dangerously-skip-permissions ""` → `transcript.jsonl` written → `bin/check` appends one TSV row. ### Grid run (full measurement) Once environment is stable (plugins cache fresh, hooks verified, fixture built): ```bash # Run the full grid (12 scenarios × 2 tiers × N reps) in parallel, monitoring as they complete for scenario in P1-L1-execution N1-L1-execution P2-L2-execution N2-L2-execution P3-L3-execution N3-L3-execution; do for model in haiku sonnet; do bin/run "$scenario" "$model" /tmp/adr-eval-c-grid --reps 1 --results /tmp/adr-eval-c-grid/results.tsv & done done wait cat /tmp/adr-eval-c-grid/results.tsv ``` ### Per-level pass bars (baseline expectations) Eval C does not collapse to a single score. Results are reported **per level × per tier**, with discrimination measured by the **negative scenarios' axis (b) results** (unneeded ADR creation): | Level | Positive (P*-L*): should trigger | Negative (N*-L*): should NOT create/assert | Comment | |-------|---------------------------|------------------------|---------| | L1 (Explicit) | Expected: PASS both tiers (find/create) | Expected: PASS both tiers (no unneeded create) | Explicit form + vocabulary are strongest signals for both polarity | | L2 (Moderate) | Expected: PASS sonnet, may degrade haiku | Expected: PASS sonnet, may degrade haiku | Vocabulary inference is harder at L2 | | L3 (Conceptual) | May degrade on both tiers (miss intent) | May degrade on both tiers (harder to resist recording a "decision" at intent level) | Pure intent is hardest for both directions | **Axis (a) consultation rate** (informational, not pass/fail): - Report alongside axis (b) results. Model consulting the ADR system for negatives is acceptable; don't penalize cheap, deterministic lookups. The discrimination is in axis (b): unneeded ADR creation. - The checker also records `cited-adr:yes/no` per negative (informational). Truthful citation of a governing ADR while documenting/testing/auditing is CORRECT behavior — the negatives sit deliberately in ADR-covered domains. Distinguishing a true citation from a false constraint claim needs semantic judgment; a keyword check fails correct answers, so citation is never a FAIL. If false-citation ever needs to be measured, that requires a narrow judge, not a regex. **Threshold hypothesis** (for post-baseline discussion, not part of first run): - Sonnet likely passes negatives reliably through L2 (no false create/cite), may struggle L3. - Haiku likely passes L1 negatives consistently, may over-trigger (create/assert) at L2/L3. - Negatives are the core measurement; if a tier fails negatives, it means over-recording — wording iteration must then address restraint (when NOT to create), not just trigger enhancement. Evaluate the **per-level grid** (A:consultation rates + B:pass/fail) first. Summary threshold only if results show a clear pattern (e.g., "PASS positives through L2, PASS negatives through L2, PASS with consultation rate Y%"). ### Optimizing with `/autoresearch` (later stage) Not part of building this harness. When wording iteration starts: 1. **Freeze the run-set**: results and scenarios are locked as baseline. 2. **Iterate only on wording** (SKILL.md, hook note, CLAUDE.md sections) — not checker, fixtures, scenarios, or judge rubric. 3. **Reduced inner loop**: target failing cells + one passing control per tier per level. 4. **Parallel runs**: drive per-cell `bin/run` scripts in the background (each cell fully independent). 5. **Re-run full grid** once wording stabilizes. 6. **If run-set degradation occurs**, stop iteration, document the contamination, and **switch all future work to the reserve-set** for measurement. See `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` for full discipline. ## Reserve set protection Run-set and reserve-set are **structurally identical** (3 levels, P/N pairs on same target ADRs) but **different scenarios** so they are not interchangeable for measurement: - **Before any wording iteration**: run-set is the measurement set. - **After first iteration against run-set results**: run-set is contaminated (wording tuned to it). Reserve-set becomes the fresh measurement surface; run-set results are historical only. - **Lock in writing**: when a change triggers an iteration decision, document that fact in a vault note or project comment so future reviewers know which set is the current measurement baseline. The reserve-set scenarios and fixtures are never read informally, never "tried out" before being used for grid runs — same held-out discipline as the run-set. ## Layout | Path | What | | --- | --- | | `fixture/project/` | Python async job queue, 6-ADR history | | `scenarios/P1-L1-execution.md`, `N1-L1-execution.md`, ... | run-set prompts (6 scenarios) | | `scenarios-reserve/P4-L1-notifications.md`, ... | reserve-set prompts (6 scenarios) | | `bin/sandbox ` | fresh git-initialized sandbox; all scenarios use same fixture base | | `bin/run [--reps N] [--results FILE]` | headless runner (the ONLY valid execution mode) | | `bin/check [--tsv ]` | two-axis checker with negative-scenario support; exit 0/1; TSV for autoresearch | | `bin/self-test [workdir]` | model-free both-directions harness validation (fabricates transcripts; never runs Task blocks) | | `judge-rubric.md` | frozen rubric for positive-scenario axis (b) LLM judge fallback | ## Why headless-only (no Agent-tool subagents) Eval B (Eval A, actually — the in-session subagent eval) uses in-session subagents because it measures *prompted skill-execution*. Eval C measures **unprompted detection**, so each model under test must get a fresh `SessionStart` hook context. In-session subagents inherit the parent session and never see a fresh hook. `bin/run` spawns each rep as a fresh `claude -p` with cwd = sandbox; the transcript's `system/hook_response` event proves the hook fired. (Verified 2026-07-03 by eval-b author: haiku quoted the hook note verbatim in a sandbox session.) ## Fixture regeneration The fixture's `graphify-out/` is **not included** (Eval C doesn't measure graph reach; it measures cue explicitness). The ADR history was generated with the plugin's own CLIs (`bin/adr-init`, `bin/adr-new`) so frontmatter and index match the shipped format, including the mechanically-superseded 0001→0002 pair. To rebuild ADRs from scratch (not normally needed): ```bash FIXTURE="plugins/os-adr/eval-c/fixture/project" rm -rf "$FIXTURE/docs/adr" ruby plugins/os-adr/bin/adr-init --root "$FIXTURE" --template nygard # Then re-create each ADR using bin/adr-new as this README documents ``` ## Adding a scenario (later stage) Add `scenarios/X.md` (or `scenarios-reserve/X.md`): - Task block + checker metadata (target ADR, topic pattern, axis-b fallback description) - Scenario class in `bin/check` (Ruby Scenario subclass with axis-b logic) - Extend `bin/sandbox` path list - Extend `bin/self-test` both directions (perfect + over-trigger for new scenario) - Document in per-level grid above if adding a level - Keep the held-out rule: the new Task block gets no informal trial runs ## Verification before first grid run ```bash # 1. Smoke-test the fixture and hooks cd plugins/os-adr/eval-c/fixture/project && git status # 2. Verify self-test passes (model-free, no tokens spent) plugins/os-adr/eval-c/bin/self-test /tmp/adr-eval-c-test # 3. Verify one cell runs headless plugins/os-adr/eval-c/bin/run P1-L1-execution haiku /tmp/adr-eval-c-test-run --reps 1 # 4. Check the TSV cat /tmp/adr-eval-c-test-run/results.tsv ``` All must pass before running the first grid.