cc-os/plugins/os-adr/eval-c/README.md

# os-adr Eval C — ambiguity-ladder discrimination eval (held-out)

_Last updated: 2026-07-06 — harness built and self-tested (all scenarios pass model-free checks)._

Measures **generalization across decreasing cue explicitness** — does the model correctly
trigger `/os-adr:find` or `/os-adr:create` as architectural decision vocabulary becomes
less explicit, and equally important, **does it correctly NOT trigger when no decided
constraint is in play**? This is the discrimination challenge: both false positives
(over-trigger) and false negatives (miss the decision) are failures.

Eval B (unprompted-behavior baseline) optimized its wording against its own grid for 5
iterations (2026-07-04), so its 8/8 (sonnet) / 7/8 (haiku) are training-set scores. Eval C
measures whether that learned behavior generalizes to **different project, different target
ADRs, and most critically, paired negative scenarios that ensure the model isn't just learning
"always trigger"**.

> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
> transcripts and never uses the Task blocks. Informal trials contaminate the measurement.

## Measurement discipline

This eval is a **frozen measurement pass** — it runs once with a baseline grid. The moment
anyone iterates wording against the run-set results, this set is contaminated; future wording
work **must switch to the reserve-set** for measurement. See "Reserve set protection" below.

## Design: ambiguity ladder with paired positive/negative

### The ladder (3 levels)

Each level decreases explicit architectural decision cues:

- **Level 1 (Explicit):** File paths named, implementation vocabulary explicit ("timeout",
  "retry", "backoff", "exponential"), decision nature overt.
- **Level 2 (Moderate):** Relevant vocabulary present but no file paths or explicit decision
  framing; model must recognize architecture-level significance.
- **Level 3 (Conceptual):** Intent-framed only (what outcome is needed), zero ADR vocabulary,
  no file paths. Model must recognize that this is an architectural concern.

### Paired positive/negative at every level

For each level, **two scenarios that differ only in whether a decided constraint is in play**:

- **Positive (P)**: A decided constraint IS applicable → consulting `/os-adr:find` and/or
  recording with `/os-adr:create` is correct.
- **Negative (N)**: Matched framing (same vocabulary level, same form — question or intent)
  but NO Accepted ADR constrains the task → consulting/creating an ADR is an over-trigger.

The negative scenario is the discrimination test: wording tuned only for sensitivity ("always
trigger") will fail negatives. Precision (correct non-triggering) is as important as recall.

### Non-monotonic difficulty expected

Difficulty does NOT increase uniformly L1 → L2 → L3. An explicit cue (L1) might be harder to
miss than a conceptual one (L3), depending on model tier and wording. Per `running-autoresearch-skill-evals.md`, each level is validated independently and results reported in a per-level grid, not collapsed to a single threshold.

## Two target ADR pairs, independent measurement

**Run-set** (6 scenarios, frozen for measurement):
- Target ADR 1: 0002 — "Job timeouts after 5 minutes with exponential backoff, max 3 attempts" (Accepted)
- Target ADR 2: 0001 → 0002 (Superseded) for distractor validation
- Scenarios: P1-L1, N1-L1, P2-L2, N2-L2, P3-L3, N3-L3 (all about job execution timeout/retry)

**Reserve-set** (6 scenarios, held until run-set is contaminated by wording iteration):
- Target ADR 2: 0003 — "Route all job notifications through central NotificationService" (Accepted)
- Scenarios: P4-L1, N4-L1, P5-L2, N5-L2, P6-L3, N6-L3 (all about notification routing)

Both sets are valid measurement surfaces; the split is for the protection protocol, not
quality difference.

## Scenario framing discipline (critical)

The CLAUDE.md instruction carried verbatim in the fixture has two triggers:
- Semantic: "architecture-level choice → run `/os-adr:find`"
- Mechanical: "before your first edit to any existing source/config file → run `/os-adr:find`"

**Negative scenarios avoid the mechanical trigger**: they're framed as pure questions or new
module designs, so trigger #2 never fires. Then "no ADR touch" is a clean, valid over-trigger
signal. If a negative scenario edits an existing file, the over-trigger criteria shift to
"created an ADR claiming a phantom constraint," requiring judge fallback.

## Fixture

**Project:** `fixture/project/` — small Python async job queue system (`taskq`).

**ADR history:** 6 ADRs (2026-02-01 to 2026-03-01), generated with plugin CLIs:
- 0001: "Job timeout at 10s, simple retry" (Superseded)
- 0002: "Job timeout at 5 min, exponential backoff, max 3 attempts" (Accepted, supersedes 0001)
- 0003: "Route notifications through central service" (Accepted)
- 0004: "Persist queue to SQLite" (Accepted)
- 0005: "Failed jobs escalate to human review queue" (Accepted)
- 0006: "All job state transitions logged to audit trail" (Accepted)

Each ADR has `affected-paths` set (e.g., 0002 → [taskq/execution.py, taskq/queue.py]) so
`bin/adr-find` can validate path-matching behavior if needed. `.graphifyignore` excludes
markdown, so any future graph build (not part of this eval) would be pure-AST / reproducible.

## Two-axis scoring

- **Axis (a) — consultation trigger**
  - **Positive scenarios**: mechanical PASS iff any `tool_use` block invoked an `os-adr:*`
    skill, `bin/adr-*` CLI, or touched `docs/adr/`.
  - **Negative scenarios**: INFORMATIONAL ONLY (recorded as `A:yes` or `A:no`, does NOT
    determine pass/fail). Consultation is acceptable per CLAUDE.md's "cheap, deterministic CLI
    call" endorsement. The discrimination measurement is not whether the model consulted, but
    whether it **recorded a false decision** or **asserted a non-existent constraint**.
  - Hook context (`SessionStart [os-adr]` event) is a precondition: missing hook → FAIL both
    axes.

- **Axis (b) — specific correctness**
  - **Positive scenarios** (write-trigger):
    - Mechanical PASS when a new `docs/adr/0007-*.md` exists and its text matches the
      scenario's topic pattern (regex per scenario class).
    - Otherwise: judge fallback (haiku, `judge-rubric.md`, reads only final message;
      stubbed via `ADR_EVAL_C_JUDGE_CMD` for self-tests).
  - **Negative scenarios** (non-decisional tasks, investigate/test/debug framing):
    - Mechanical FAIL if a new ADR file was created (over-trigger: creating a decision record
      for a non-decisional task). This is the ONLY fail condition.
    - Citation of existing ADRs in the final message is recorded as informational
      `cited-adr:yes/no`, never a FAIL — the negatives sit deliberately in ADR-covered
      domains, and truthfully citing a governing ADR while documenting/testing/auditing is
      correct behavior. Separating true citations from false constraint claims needs
      semantic judgment; a keyword match fails correct answers.
    - Otherwise: PASS.

**Overall PASS**:
- **Positive scenarios**: both axes PASS.
- **Negative scenarios**: axis (b) PASS only (axis a is informational, doesn't affect pass/fail).

TSV format: `scenario  model  A:yes|no-or-PASS|FAIL  B:PASS|FAIL  PASS|FAIL  reasons`

## Running the evaluation

### Single cell (manual test)

```bash
eval-c/bin/sandbox P2-L2-execution /tmp/adr-eval-c-test
eval-c/bin/run P2-L2-execution haiku /tmp/adr-eval-c-test --reps 1 --results /tmp/results.tsv
cat /tmp/results.tsv
```

Each rep: fresh sandbox → `claude -p --model <tier> --output-format stream-json
--dangerously-skip-permissions "<Task block>"` → `transcript.jsonl` written → `bin/check`
appends one TSV row.

### Grid run (full measurement)

Once environment is stable (plugins cache fresh, hooks verified, fixture built):

```bash
# Run the full grid (12 scenarios × 2 tiers × N reps) in parallel, monitoring as they complete
for scenario in P1-L1-execution N1-L1-execution P2-L2-execution N2-L2-execution P3-L3-execution N3-L3-execution; do
  for model in haiku sonnet; do
    bin/run "$scenario" "$model" /tmp/adr-eval-c-grid --reps 1 --results /tmp/adr-eval-c-grid/results.tsv &
  done
done
wait
cat /tmp/adr-eval-c-grid/results.tsv
```

### Per-level pass bars (baseline expectations)

Eval C does not collapse to a single score. Results are reported **per level × per tier**,
with discrimination measured by the **negative scenarios' axis (b) results** (unneeded ADR creation):

| Level | Positive (P*-L*): should trigger | Negative (N*-L*): should NOT create/assert | Comment |
|-------|---------------------------|------------------------|---------|
| L1 (Explicit) | Expected: PASS both tiers (find/create) | Expected: PASS both tiers (no unneeded create) | Explicit form + vocabulary are strongest signals for both polarity |
| L2 (Moderate) | Expected: PASS sonnet, may degrade haiku | Expected: PASS sonnet, may degrade haiku | Vocabulary inference is harder at L2 |
| L3 (Conceptual) | May degrade on both tiers (miss intent) | May degrade on both tiers (harder to resist recording a "decision" at intent level) | Pure intent is hardest for both directions |

**Axis (a) consultation rate** (informational, not pass/fail):
- Report alongside axis (b) results. Model consulting the ADR system for negatives is acceptable;
  don't penalize cheap, deterministic lookups. The discrimination is in axis (b): unneeded ADR creation.
- The checker also records `cited-adr:yes/no` per negative (informational). Truthful citation of a
  governing ADR while documenting/testing/auditing is CORRECT behavior — the negatives sit
  deliberately in ADR-covered domains. Distinguishing a true citation from a false constraint claim
  needs semantic judgment; a keyword check fails correct answers, so citation is never a FAIL.
  If false-citation ever needs to be measured, that requires a narrow judge, not a regex.

**Threshold hypothesis** (for post-baseline discussion, not part of first run):
- Sonnet likely passes negatives reliably through L2 (no false create/cite), may struggle L3.
- Haiku likely passes L1 negatives consistently, may over-trigger (create/assert) at L2/L3.
- Negatives are the core measurement; if a tier fails negatives, it means over-recording —
  wording iteration must then address restraint (when NOT to create), not just trigger enhancement.

Evaluate the **per-level grid** (A:consultation rates + B:pass/fail) first. Summary threshold only
if results show a clear pattern (e.g., "PASS positives through L2, PASS negatives through L2,
PASS with consultation rate Y%").

### Optimizing with `/autoresearch` (later stage)

Not part of building this harness. When wording iteration starts:

1. **Freeze the run-set**: results and scenarios are locked as baseline.
2. **Iterate only on wording** (SKILL.md, hook note, CLAUDE.md sections) — not checker,
   fixtures, scenarios, or judge rubric.
3. **Reduced inner loop**: target failing cells + one passing control per tier per level.
4. **Parallel runs**: drive per-cell `bin/run` scripts in the background (each cell fully
   independent).
5. **Re-run full grid** once wording stabilizes.
6. **If run-set degradation occurs**, stop iteration, document the contamination, and
   **switch all future work to the reserve-set** for measurement.

See `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` for full discipline.

## Reserve set protection

Run-set and reserve-set are **structurally identical** (3 levels, P/N pairs on same target
ADRs) but **different scenarios** so they are not interchangeable for measurement:

- **Before any wording iteration**: run-set is the measurement set.
- **After first iteration against run-set results**: run-set is contaminated (wording tuned to
  it). Reserve-set becomes the fresh measurement surface; run-set results are historical
  only.
- **Lock in writing**: when a change triggers an iteration decision, document that fact in a
  vault note or project comment so future reviewers know which set is the current measurement
  baseline.

The reserve-set scenarios and fixtures are never read informally, never "tried out" before
being used for grid runs — same held-out discipline as the run-set.

## Layout

| Path | What |
| --- | --- |
| `fixture/project/` | Python async job queue, 6-ADR history |
| `scenarios/P1-L1-execution.md`, `N1-L1-execution.md`, ... | run-set prompts (6 scenarios) |
| `scenarios-reserve/P4-L1-notifications.md`, ... | reserve-set prompts (6 scenarios) |
| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; all scenarios use same fixture base |
| `bin/run <Sn> <model> <workdir> [--reps N] [--results FILE]` | headless runner (the ONLY valid execution mode) |
| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker with negative-scenario support; exit 0/1; TSV for autoresearch |
| `bin/self-test [workdir]` | model-free both-directions harness validation (fabricates transcripts; never runs Task blocks) |
| `judge-rubric.md` | frozen rubric for positive-scenario axis (b) LLM judge fallback |

## Why headless-only (no Agent-tool subagents)

Eval B (Eval A, actually — the in-session subagent eval) uses in-session subagents because
it measures *prompted skill-execution*. Eval C measures **unprompted detection**, so each
model under test must get a fresh `SessionStart` hook context. In-session subagents inherit
the parent session and never see a fresh hook. `bin/run` spawns each rep as a fresh
`claude -p` with cwd = sandbox; the transcript's `system/hook_response` event proves the hook
fired. (Verified 2026-07-03 by eval-b author: haiku quoted the hook note verbatim in a
sandbox session.)

## Fixture regeneration

The fixture's `graphify-out/` is **not included** (Eval C doesn't measure graph reach; it
measures cue explicitness). The ADR history was generated with the plugin's own CLIs
(`bin/adr-init`, `bin/adr-new`) so frontmatter and index match the shipped format, including
the mechanically-superseded 0001→0002 pair.

To rebuild ADRs from scratch (not normally needed):

```bash
FIXTURE="plugins/os-adr/eval-c/fixture/project"
rm -rf "$FIXTURE/docs/adr"
ruby plugins/os-adr/bin/adr-init --root "$FIXTURE" --template nygard
# Then re-create each ADR using bin/adr-new as this README documents
```

## Adding a scenario (later stage)

Add `scenarios/X.md` (or `scenarios-reserve/X.md`):
- Task block + checker metadata (target ADR, topic pattern, axis-b fallback description)
- Scenario class in `bin/check` (Ruby Scenario subclass with axis-b logic)
- Extend `bin/sandbox` path list
- Extend `bin/self-test` both directions (perfect + over-trigger for new scenario)
- Document in per-level grid above if adding a level
- Keep the held-out rule: the new Task block gets no informal trial runs

## Verification before first grid run

```bash
# 1. Smoke-test the fixture and hooks
cd plugins/os-adr/eval-c/fixture/project && git status

# 2. Verify self-test passes (model-free, no tokens spent)
plugins/os-adr/eval-c/bin/self-test /tmp/adr-eval-c-test

# 3. Verify one cell runs headless
plugins/os-adr/eval-c/bin/run P1-L1-execution haiku /tmp/adr-eval-c-test-run --reps 1

# 4. Check the TSV
cat /tmp/adr-eval-c-test-run/results.tsv
```

All must pass before running the first grid.