New global plugin with one SessionStart hook running all checks in-process
(hooks/checks.py, check(ctx) -> CheckResult, single registry). Three states:
ok (silent), note (near-zero-token line, never suppressed), warn (one
aggregated banner, once-per-day snooze + permanent suppress in gitignored
per-project .cc-os/). Initial checks: subagent-model-env-override (catches
the WS1 Cluster 1 incident deterministically), adr-system-present (verbatim
port of os-adr's hook, legacy .os-adr/suppress honored), vault-hub-note-present.
os-adr's own SessionStart hook moved here atomically (its hooks.json now
empty; session_start.py stays as wording source of record). Built via
OpenSpec change add-os-status-plugin; 36 model-free tests; smoke-tested
2026-07-06 incl. env-override canary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First full run of the Eval C discrimination harness (3-level ambiguity ladder,
6-scenario run-set, 3 reps/cell, both sonnet+haiku): 36 headless reps, 12/12
cells PASS at ≥2/3 majority. Haiku 18/18 reps, sonnet 17/18 (P1-L1 rep 1: final
message citing only 0005, not governing 0002 — a boundary artifact, not a
behavioral miss). Zero over-trigger ADR creation on any negative at any level;
all positive passes via governing-ADR citation (16 reps) or judge fallback (1).
Two harness defects fixed before running the grid: (1) bin/run pipefail bug that
aborted remaining reps after first FAIL (changed checker invocation to || true);
(2) positive axis-b previously required ADR creation only, contradicting the
README's design statement that "consulting and/or recording is correct" — run-set
positives sit in already-decided territory, so find→cite→comply is the correct
behavior. Implemented three-branch axis-b check: created (new ADR file),
cited-governing (final message cites the governing Accepted ADR), or judge
fallback. Added two self-test guards to verify the compliance path.
Scenario metadata updated to document the corrected axis-b checker behavior
(cited-governing as a valid pass branch). TSV reasons column now records the
pass path via `B-via:` field for positives.
Verdict: Eval B wording generalizes to a new fixture, new language, new domain,
and decreasing cue explicitness. Grid ceiling not found; next signal is
production (rollout + IRL session audits). Run-set baseline locked; wording
tuning moves measurement to reserve-set.
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set
after five iterations; subsequent W3 stability testing confirmed
intermittent flicker but acceptable performance. As Eval B is now
contaminated by iteration, we need a held-out measurement set to
assess generalization to new projects and cues.
Eval C measures whether learned behavior generalizes across
decreasing cue explicitness (explicit → moderate → conceptual
framing) and, critically, whether the model correctly avoids
false positives (over-triggering) when no Accepted ADR is in play.
Six paired scenarios (positive/negative at each level) run against
a job-execution domain fixture; a frozen reserve-set (notifications
domain) becomes the measurement set if anyone tunes wording against
the run-set.
New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox
scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased
CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve-
set), README.md (measurement discipline + design), judge-rubric.md.
Model-free self-test passing. First real grid run is a pending
decision.
Five-iteration /autoresearch wording experiment against the Eval B baseline
(haiku 0/8, sonnet 5/8). Checker/fixtures/scenarios/rubric frozen; wording
surfaces: hook PRESENT_NOTE, find/create SKILL.mds, fixture CLAUDE.md
(declared a surface upfront — it is the real-project adoption template).
- Hook note + skill descriptions: when->then trigger-conditioned phrasing
(H1 confirmed, both tiers)
- find skill: explicit-unconditional reversal->supersede offer where find's
output is on screen (fixed sonnet W3)
- Mechanical lower-tier trigger: before first edit to any existing file,
run find on those paths; additions count (fixed haiku R1/R4)
- New eval-b/fixture/project/CLAUDE.md carrying the same rules
Final grid: sonnet 8/8, haiku 7/8 (W3 axis-b judge flicker). Open: channel
ablation (hook vs CLAUDE.md) not run; R4-nograph now passes both tiers.
Hypothesis->result map: vault note os-adr-eval-b-wording-experiment-hypotheses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Remove explicit name: frontmatter from os-adr and os-doc-hygiene SKILL.mds
so slash commands register namespaced (/os-adr:find, not bare /find)
- Add bin/refresh-plugins to refresh stale local plugin caches
- CLAUDE.md: cache-refresh procedure, naming-convention pointer, mandate
reading the vault autoresearch-eval howto before the next eval
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eval A grid ran clean across all scenarios/tiers. Eval B grid (1 rep/cell)
shows a real gap: haiku never unprompted-consults the ADR system in any
scenario, while sonnet passes 5/8 (misses W3 write-trigger and R1 direct-
conflict retrieval). Updates CLAUDE.md component status and the eval-b
README so the next session picks up from here instead of re-running blind.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jared