16 KiB

Raw Permalink Blame History

os-adr Eval C — ambiguity-ladder discrimination eval (held-out)

Last updated: 2026-07-06 — harness built and self-tested (all scenarios pass model-free checks).

Measures generalization across decreasing cue explicitness — does the model correctly trigger /os-adr:find or /os-adr:create as architectural decision vocabulary becomes less explicit, and equally important, does it correctly NOT trigger when no decided constraint is in play? This is the discrimination challenge: both false positives (over-trigger) and false negatives (miss the decision) are failures.

Eval B (unprompted-behavior baseline) optimized its wording against its own grid for 5 iterations (2026-07-04), so its 8/8 (sonnet) / 7/8 (haiku) are training-set scores. Eval C measures whether that learned behavior generalizes to different project, different target ADRs, and most critically, paired negative scenarios that ensure the model isn't just learning "always trigger".

HELD-OUT — read this first. The ## Task blocks in scenarios/*.md and scenarios-reserve/*.md must never be pasted into an interactive session or informally "tried out". The only permitted non-grid execution is bin/self-test, which fabricates transcripts and never uses the Task blocks. Informal trials contaminate the measurement.

Measurement discipline

This eval is a frozen measurement pass — it runs once with a baseline grid. The moment anyone iterates wording against the run-set results, this set is contaminated; future wording work must switch to the reserve-set for measurement. See "Reserve set protection" below.

Design: ambiguity ladder with paired positive/negative

The ladder (3 levels)

Each level decreases explicit architectural decision cues:

Level 1 (Explicit): File paths named, implementation vocabulary explicit ("timeout", "retry", "backoff", "exponential"), decision nature overt.
Level 2 (Moderate): Relevant vocabulary present but no file paths or explicit decision framing; model must recognize architecture-level significance.
Level 3 (Conceptual): Intent-framed only (what outcome is needed), zero ADR vocabulary, no file paths. Model must recognize that this is an architectural concern.

Paired positive/negative at every level

For each level, two scenarios that differ only in whether a decided constraint is in play:

Positive (P): A decided constraint IS applicable → consulting /os-adr:find and/or recording with /os-adr:create is correct.
Negative (N): Matched framing (same vocabulary level, same form — question or intent) but NO Accepted ADR constrains the task → consulting/creating an ADR is an over-trigger.

The negative scenario is the discrimination test: wording tuned only for sensitivity ("always trigger") will fail negatives. Precision (correct non-triggering) is as important as recall.

Non-monotonic difficulty expected

Difficulty does NOT increase uniformly L1 → L2 → L3. An explicit cue (L1) might be harder to miss than a conceptual one (L3), depending on model tier and wording. Per running-autoresearch-skill-evals.md, each level is validated independently and results reported in a per-level grid, not collapsed to a single threshold.

Two target ADR pairs, independent measurement

Run-set (6 scenarios, frozen for measurement):

Target ADR 1: 0002 — "Job timeouts after 5 minutes with exponential backoff, max 3 attempts" (Accepted)
Target ADR 2: 0001 → 0002 (Superseded) for distractor validation
Scenarios: P1-L1, N1-L1, P2-L2, N2-L2, P3-L3, N3-L3 (all about job execution timeout/retry)

Reserve-set (6 scenarios, held until run-set is contaminated by wording iteration):

Target ADR 2: 0003 — "Route all job notifications through central NotificationService" (Accepted)
Scenarios: P4-L1, N4-L1, P5-L2, N5-L2, P6-L3, N6-L3 (all about notification routing)

Both sets are valid measurement surfaces; the split is for the protection protocol, not quality difference.

Scenario framing discipline (critical)

The CLAUDE.md instruction carried verbatim in the fixture has two triggers:

Semantic: "architecture-level choice → run /os-adr:find"
Mechanical: "before your first edit to any existing source/config file → run /os-adr:find"

Negative scenarios avoid the mechanical trigger: they're framed as pure questions or new module designs, so trigger #2 never fires. Then "no ADR touch" is a clean, valid over-trigger signal. If a negative scenario edits an existing file, the over-trigger criteria shift to "created an ADR claiming a phantom constraint," requiring judge fallback.

Fixture

Project: fixture/project/ — small Python async job queue system (taskq).

ADR history: 6 ADRs (2026-02-01 to 2026-03-01), generated with plugin CLIs:

0001: "Job timeout at 10s, simple retry" (Superseded)
0002: "Job timeout at 5 min, exponential backoff, max 3 attempts" (Accepted, supersedes 0001)
0003: "Route notifications through central service" (Accepted)
0004: "Persist queue to SQLite" (Accepted)
0005: "Failed jobs escalate to human review queue" (Accepted)
0006: "All job state transitions logged to audit trail" (Accepted)

Each ADR has affected-paths set (e.g., 0002 → [taskq/execution.py, taskq/queue.py]) so bin/adr-find can validate path-matching behavior if needed. .graphifyignore excludes markdown, so any future graph build (not part of this eval) would be pure-AST / reproducible.

Two-axis scoring

Axis (a) — consultation trigger
- Positive scenarios: mechanical PASS iff any tool_use block invoked an os-adr:* skill, bin/adr-* CLI, or touched docs/adr/.
- Negative scenarios: INFORMATIONAL ONLY (recorded as A:yes or A:no, does NOT determine pass/fail). Consultation is acceptable per CLAUDE.md's "cheap, deterministic CLI call" endorsement. The discrimination measurement is not whether the model consulted, but whether it recorded a false decision or asserted a non-existent constraint.
- Hook context (SessionStart [os-adr] event) is a precondition: missing hook → FAIL both axes.
Axis (b) — specific correctness
- Positive scenarios (decided-constraint-applies): PASS via any ONE of, in order:
  1. created — a new docs/adr/NNNN-*.md exists matching the scenario's topic pattern.
  2. cited-governing — the final message cites the governing Accepted ADR by id (P1→0002, P2→0005, P3→0002/0005/0006; reserve P4–P6→0003). The positives sit in already-decided territory (each task restates what an Accepted ADR already decided), so per the shipped plugin wording the correct behavior is find → cite → comply, NOT create a duplicate ADR. This branch conforms the checker to this README's design statement ("consulting /os-adr:find and/or recording with /os-adr:create is correct"); the original checker required creation/proposal only, which failed exactly the behavior the negatives reward (fixed 2026-07-06 after 1 spent rep, before the grid). The TSV reasons column records which branch passed (B-via:created|cited-governing|judge-proposed).
  3. judge fallback (haiku, judge-rubric.md — frozen, unchanged — reads only the final message; stubbed via ADR_EVAL_C_JUDGE_CMD for self-tests).
- Negative scenarios (non-decisional tasks, investigate/test/debug framing):
  - Mechanical FAIL if a new ADR file was created (over-trigger: creating a decision record for a non-decisional task). This is the ONLY fail condition.
  - Citation of existing ADRs in the final message is recorded as informational cited-adr:yes/no, never a FAIL — the negatives sit deliberately in ADR-covered domains, and truthfully citing a governing ADR while documenting/testing/auditing is correct behavior. Separating true citations from false constraint claims needs semantic judgment; a keyword match fails correct answers.
  - Otherwise: PASS.

Overall PASS:

Positive scenarios: both axes PASS.
Negative scenarios: axis (b) PASS only (axis a is informational, doesn't affect pass/fail).

TSV format: scenario model A:yes|no-or-PASS|FAIL B:PASS|FAIL PASS|FAIL reasons

Running the evaluation

Single cell (manual test)

eval-c/bin/sandbox P2-L2-execution /tmp/adr-eval-c-test
eval-c/bin/run P2-L2-execution haiku /tmp/adr-eval-c-test --reps 1 --results /tmp/results.tsv
cat /tmp/results.tsv

Each rep: fresh sandbox → claude -p --model <tier> --output-format stream-json --dangerously-skip-permissions "<Task block>" → transcript.jsonl written → bin/check appends one TSV row.

Grid run (full measurement)

Once environment is stable (plugins cache fresh, hooks verified, fixture built):

# Run the full grid (12 scenarios × 2 tiers × N reps) in parallel, monitoring as they complete
for scenario in P1-L1-execution N1-L1-execution P2-L2-execution N2-L2-execution P3-L3-execution N3-L3-execution; do
  for model in haiku sonnet; do
    bin/run "$scenario" "$model" /tmp/adr-eval-c-grid --reps 1 --results /tmp/adr-eval-c-grid/results.tsv &
  done
done
wait
cat /tmp/adr-eval-c-grid/results.tsv

Per-level pass bars (baseline expectations)

Eval C does not collapse to a single score. Results are reported per level × per tier, with discrimination measured by the negative scenarios' axis (b) results (unneeded ADR creation):

Level	Positive (P-L): should trigger	Negative (N-L): should NOT create/assert	Comment
L1 (Explicit)	Expected: PASS both tiers (find/create)	Expected: PASS both tiers (no unneeded create)	Explicit form + vocabulary are strongest signals for both polarity
L2 (Moderate)	Expected: PASS sonnet, may degrade haiku	Expected: PASS sonnet, may degrade haiku	Vocabulary inference is harder at L2
L3 (Conceptual)	May degrade on both tiers (miss intent)	May degrade on both tiers (harder to resist recording a "decision" at intent level)	Pure intent is hardest for both directions

Axis (a) consultation rate (informational, not pass/fail):

Report alongside axis (b) results. Model consulting the ADR system for negatives is acceptable; don't penalize cheap, deterministic lookups. The discrimination is in axis (b): unneeded ADR creation.
The checker also records cited-adr:yes/no per negative (informational). Truthful citation of a governing ADR while documenting/testing/auditing is CORRECT behavior — the negatives sit deliberately in ADR-covered domains. Distinguishing a true citation from a false constraint claim needs semantic judgment; a keyword check fails correct answers, so citation is never a FAIL. If false-citation ever needs to be measured, that requires a narrow judge, not a regex.

Threshold hypothesis (for post-baseline discussion, not part of first run):

Sonnet likely passes negatives reliably through L2 (no false create/cite), may struggle L3.
Haiku likely passes L1 negatives consistently, may over-trigger (create/assert) at L2/L3.
Negatives are the core measurement; if a tier fails negatives, it means over-recording — wording iteration must then address restraint (when NOT to create), not just trigger enhancement.

Evaluate the per-level grid (A:consultation rates + B:pass/fail) first. Summary threshold only if results show a clear pattern (e.g., "PASS positives through L2, PASS negatives through L2, PASS with consultation rate Y%").

Optimizing with `/autoresearch` (later stage)

Not part of building this harness. When wording iteration starts:

Freeze the run-set: results and scenarios are locked as baseline.
Iterate only on wording (SKILL.md, hook note, CLAUDE.md sections) — not checker, fixtures, scenarios, or judge rubric.
Reduced inner loop: target failing cells + one passing control per tier per level.
Parallel runs: drive per-cell bin/run scripts in the background (each cell fully independent).
Re-run full grid once wording stabilizes.
If run-set degradation occurs, stop iteration, document the contamination, and switch all future work to the reserve-set for measurement.

See ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md for full discipline.

Reserve set protection

Run-set and reserve-set are structurally identical (3 levels, P/N pairs on same target ADRs) but different scenarios so they are not interchangeable for measurement:

Before any wording iteration: run-set is the measurement set.
After first iteration against run-set results: run-set is contaminated (wording tuned to it). Reserve-set becomes the fresh measurement surface; run-set results are historical only.
Lock in writing: when a change triggers an iteration decision, document that fact in a vault note or project comment so future reviewers know which set is the current measurement baseline.

The reserve-set scenarios and fixtures are never read informally, never "tried out" before being used for grid runs — same held-out discipline as the run-set.

Layout

Path	What
`fixture/project/`	Python async job queue, 6-ADR history
`scenarios/P1-L1-execution.md`, `N1-L1-execution.md`, ...	run-set prompts (6 scenarios)
`scenarios-reserve/P4-L1-notifications.md`, ...	reserve-set prompts (6 scenarios)
`bin/sandbox <Sn> <dest>`	fresh git-initialized sandbox; all scenarios use same fixture base
`bin/run <Sn> <model> <workdir> [--reps N] [--results FILE]`	headless runner (the ONLY valid execution mode)
`bin/check <Sn> <sandbox> [--tsv <model>]`	two-axis checker with negative-scenario support; exit 0/1; TSV for autoresearch
`bin/self-test [workdir]`	model-free both-directions harness validation (fabricates transcripts; never runs Task blocks)
`judge-rubric.md`	frozen rubric for positive-scenario axis (b) LLM judge fallback

Why headless-only (no Agent-tool subagents)

Eval B (Eval A, actually — the in-session subagent eval) uses in-session subagents because it measures prompted skill-execution. Eval C measures unprompted detection, so each model under test must get a fresh SessionStart hook context. In-session subagents inherit the parent session and never see a fresh hook. bin/run spawns each rep as a fresh claude -p with cwd = sandbox; the transcript's system/hook_response event proves the hook fired. (Verified 2026-07-03 by eval-b author: haiku quoted the hook note verbatim in a sandbox session.)

Fixture regeneration

The fixture's graphify-out/ is not included (Eval C doesn't measure graph reach; it measures cue explicitness). The ADR history was generated with the plugin's own CLIs (bin/adr-init, bin/adr-new) so frontmatter and index match the shipped format, including the mechanically-superseded 0001→0002 pair.

To rebuild ADRs from scratch (not normally needed):

FIXTURE="plugins/os-adr/eval-c/fixture/project"
rm -rf "$FIXTURE/docs/adr"
ruby plugins/os-adr/bin/adr-init --root "$FIXTURE" --template nygard
# Then re-create each ADR using bin/adr-new as this README documents

Adding a scenario (later stage)

Add scenarios/X.md (or scenarios-reserve/X.md):

Task block + checker metadata (target ADR, topic pattern, axis-b fallback description)
Scenario class in bin/check (Ruby Scenario subclass with axis-b logic)
Extend bin/sandbox path list
Extend bin/self-test both directions (perfect + over-trigger for new scenario)
Document in per-level grid above if adding a level
Keep the held-out rule: the new Task block gets no informal trial runs

Verification before first grid run

# 1. Smoke-test the fixture and hooks
cd plugins/os-adr/eval-c/fixture/project && git status

# 2. Verify self-test passes (model-free, no tokens spent)
plugins/os-adr/eval-c/bin/self-test /tmp/adr-eval-c-test

# 3. Verify one cell runs headless
plugins/os-adr/eval-c/bin/run P1-L1-execution haiku /tmp/adr-eval-c-test-run --reps 1

# 4. Check the TSV
cat /tmp/adr-eval-c-test-run/results.tsv

All must pass before running the first grid.

16 KiB Raw Permalink Blame History Unescape Escape