- **Level 2 (Moderate):** Relevant vocabulary present but no file paths or explicit decision
framing; model must recognize architecture-level significance.
- **Level 3 (Conceptual):** Intent-framed only (what outcome is needed), zero ADR vocabulary,
no file paths. Model must recognize that this is an architectural concern.
### Paired positive/negative at every level
For each level, **two scenarios that differ only in whether a decided constraint is in play**:
- **Positive (P)**: A decided constraint IS applicable → consulting `/os-adr:find` and/or
recording with `/os-adr:create` is correct.
- **Negative (N)**: Matched framing (same vocabulary level, same form — question or intent)
but NO Accepted ADR constrains the task → consulting/creating an ADR is an over-trigger.
The negative scenario is the discrimination test: wording tuned only for sensitivity ("always
trigger") will fail negatives. Precision (correct non-triggering) is as important as recall.
### Non-monotonic difficulty expected
Difficulty does NOT increase uniformly L1 → L2 → L3. An explicit cue (L1) might be harder to
miss than a conceptual one (L3), depending on model tier and wording. Per `running-autoresearch-skill-evals.md`, each level is validated independently and results reported in a per-level grid, not collapsed to a single threshold.
## Two target ADR pairs, independent measurement
**Run-set** (6 scenarios, frozen for measurement):
- Target ADR 1: 0002 — "Job timeouts after 5 minutes with exponential backoff, max 3 attempts" (Accepted)
| L1 (Explicit) | Expected: PASS both tiers (find/create) | Expected: PASS both tiers (no unneeded create) | Explicit form + vocabulary are strongest signals for both polarity |
| L2 (Moderate) | Expected: PASS sonnet, may degrade haiku | Expected: PASS sonnet, may degrade haiku | Vocabulary inference is harder at L2 |
| L3 (Conceptual) | May degrade on both tiers (miss intent) | May degrade on both tiers (harder to resist recording a "decision" at intent level) | Pure intent is hardest for both directions |
**Axis (a) consultation rate** (informational, not pass/fail):
- Report alongside axis (b) results. Model consulting the ADR system for negatives is acceptable;
don't penalize cheap, deterministic lookups. The discrimination is in axis (b): unneeded ADR creation.
- The checker also records `cited-adr:yes/no` per negative (informational). Truthful citation of a
governing ADR while documenting/testing/auditing is CORRECT behavior — the negatives sit
deliberately in ADR-covered domains. Distinguishing a true citation from a false constraint claim
needs semantic judgment; a keyword check fails correct answers, so citation is never a FAIL.
If false-citation ever needs to be measured, that requires a narrow judge, not a regex.
**Threshold hypothesis** (for post-baseline discussion, not part of first run):
- Sonnet likely passes negatives reliably through L2 (no false create/cite), may struggle L3.
- Haiku likely passes L1 negatives consistently, may over-trigger (create/assert) at L2/L3.
- Negatives are the core measurement; if a tier fails negatives, it means over-recording —
wording iteration must then address restraint (when NOT to create), not just trigger enhancement.
Evaluate the **per-level grid** (A:consultation rates + B:pass/fail) first. Summary threshold only
if results show a clear pattern (e.g., "PASS positives through L2, PASS negatives through L2,
PASS with consultation rate Y%").
### Optimizing with `/autoresearch` (later stage)
Not part of building this harness. When wording iteration starts:
1.**Freeze the run-set**: results and scenarios are locked as baseline.
2.**Iterate only on wording** (SKILL.md, hook note, CLAUDE.md sections) — not checker,
fixtures, scenarios, or judge rubric.
3.**Reduced inner loop**: target failing cells + one passing control per tier per level.
4.**Parallel runs**: drive per-cell `bin/run` scripts in the background (each cell fully
independent).
5.**Re-run full grid** once wording stabilizes.
6.**If run-set degradation occurs**, stop iteration, document the contamination, and
**switch all future work to the reserve-set** for measurement.
See `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` for full discipline.
## Reserve set protection
Run-set and reserve-set are **structurally identical** (3 levels, P/N pairs on same target
ADRs) but **different scenarios** so they are not interchangeable for measurement:
- **Before any wording iteration**: run-set is the measurement set.
- **After first iteration against run-set results**: run-set is contaminated (wording tuned to
it). Reserve-set becomes the fresh measurement surface; run-set results are historical
only.
- **Lock in writing**: when a change triggers an iteration decision, document that fact in a
vault note or project comment so future reviewers know which set is the current measurement
baseline.
The reserve-set scenarios and fixtures are never read informally, never "tried out" before
being used for grid runs — same held-out discipline as the run-set.
## Layout
| Path | What |
| --- | --- |
| `fixture/project/` | Python async job queue, 6-ADR history |
