diff --git a/CLAUDE.md b/CLAUDE.md index 8dd093b..ae0dd83 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,7 +101,8 @@ to those two and fix the stale doc. - Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate. - Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario — a real gap, not a harness defect); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). `R4-nograph` FAILed on both tiers as expected (degradation check, only meaningful paired with an R4 PASS — sonnet has one, haiku doesn't). Full results + observations (prompting-issue hypothesis, open question on whether in-session subagents could ever validly substitute for part of this measurement) written to the vault: [[os-adr-eval-b-grid-results-and-observations]]. - **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight. -- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; then design and run a follow-up eval iterating Eval B trigger/retrieval wording via `/autoresearch` (checker/fixtures/scenarios/rubric frozen) targeting the W3 and R1 failures on sonnet and the wholesale haiku gap. **Before designing or running that (or any) autoresearch eval, Read the vault howto `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`** — it locks the procedure (valid run mode per eval shape, cache refresh via `bin/refresh-plugins` after every wording edit, reduced parallel inner-loop grid, ~3 reps on target cells, axis-level failure reading, verify from TSV not agent prose). Baseline numbers + the 2026-07-04 confirmation run live in the vault note above (baseline reconfirmed: haiku 0/8, sonnet 5/8 failing W3/R1; W3 is an axis-b failure — consults but doesn't propose recording — so its wording target is the create skill, while R1 is an axis-a trigger failure). +- **Wording experiment complete (2026-07-04):** the follow-up `/autoresearch` loop over Eval B wording ran (5 iterations, checker/fixtures/scenarios/rubric frozen; fixture CLAUDE.md declared a wording surface upfront) and closed the gap — final full grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8 baseline; haiku's one miss is a W3 axis-b judge-boundary flicker). Winning wording shipped in `hooks/session_start.py` (PRESENT_NOTE), find/create SKILL.mds, and `eval-b/fixture/project/CLAUDE.md` (new — the trigger-phrased "Architecture decisions" section, the template for real-project adoption). Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule must live where its precondition is visible (the reversal→supersede rule in the find skill's act-on-findings step fixed W3); lower tiers need *mechanical* triggers ("before your first edit to any existing file → run `/os-adr:find` on those paths; additions count") — semantic triggers ("architecture-level choice") only reach sonnet. Open: channel ablation never run (hook vs CLAUDE.md redundancy unknown); R4-nograph now passes both tiers, so the graph-degradation check no longer differentiates. Full hypothesis→result mapping: vault note [[os-adr-eval-b-wording-experiment-hypotheses]]. **Before designing or running any autoresearch eval, Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.** +- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider. When onboarding real projects, add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`). - **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh. **Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify` diff --git a/plugins/os-adr/eval-b/fixture/project/CLAUDE.md b/plugins/os-adr/eval-b/fixture/project/CLAUDE.md new file mode 100644 index 0000000..ff58c41 --- /dev/null +++ b/plugins/os-adr/eval-b/fixture/project/CLAUDE.md @@ -0,0 +1,25 @@ +# CLAUDE.md + +`relay` is a small Ruby CLI that delivers webhook payloads to configured endpoints and +reports on delivery outcomes. Configuration lives in `config/relay.yml`; failed deliveries +that exhaust retries go to the dead-letter file. + +## Architecture decisions + +This project records architecture decisions as ADRs in `docs/adr/` (index: +`docs/adr/README.md`). + +- **When** your task involves making an architecture-level choice, or changing/replacing an + approach this codebase already uses (persistence, delivery/retry strategy, file formats, + libraries, conventions) → **first** run `/os-adr:find` to check whether a recorded decision + already covers it. +- Mechanical rule: **before your first edit to any existing source or config file**, run + `/os-adr:find` on the paths you are about to touch — one cheap, deterministic CLI call. + This applies to additions too: a new method or option can bypass a decided constraint just + as easily as a rewrite can. +- **When** you make such a choice → record it with `/os-adr:create`. +- If an Accepted ADR covers the approach you are changing, you are **reversing a recorded + decision**. A task that reverses a recorded decision is **not complete until the superseding + ADR is created** (`/os-adr:create` with `supersedes`) — implementing the change and stopping + leaves `docs/adr/` asserting the opposite of what the code does. Create the superseding ADR + (or explicitly propose it to the user) before you consider the task done. diff --git a/plugins/os-adr/hooks/session_start.py b/plugins/os-adr/hooks/session_start.py index 7fc81fa..bae5547 100644 --- a/plugins/os-adr/hooks/session_start.py +++ b/plugins/os-adr/hooks/session_start.py @@ -25,9 +25,20 @@ from typing import Optional, Tuple PRESENT_NOTE = ( "[os-adr] This project records architecture decisions in docs/adr/ " - "(index: docs/adr/README.md). Record a new decision of consequence with " - "/os-adr:create; before changing decided-on behavior, check relevant " - "decisions with /os-adr:find." + "(index: docs/adr/README.md). WHEN your task involves making an " + "architecture-level choice, or changing/replacing an approach that " + "already exists in this codebase (persistence, protocols, libraries, " + "conventions, error handling) -> FIRST run /os-adr:find to check whether " + "a decision already covers it. Mechanical rule: BEFORE your first edit " + "to any existing source or config file -> run /os-adr:find on the paths " + "you are about to touch (one cheap deterministic CLI call; additions " + "count — a new method can bypass a decided constraint). WHEN " + "you make such a choice -> record it " + "with /os-adr:create. If find shows an Accepted ADR covering the approach " + "you are changing, you are reversing a recorded decision: the task is " + "NOT COMPLETE until the superseding ADR is created with /os-adr:create " + "(supersedes: NNNN) — code changed but ADR unrecorded means the decision " + "history now asserts the opposite of what the code does." ) ABSENT_NOTE = ( diff --git a/plugins/os-adr/skills/create/SKILL.md b/plugins/os-adr/skills/create/SKILL.md index 72576d6..b8844d8 100644 --- a/plugins/os-adr/skills/create/SKILL.md +++ b/plugins/os-adr/skills/create/SKILL.md @@ -1,11 +1,20 @@ --- -description: Record an architecture decision as a correctly templated, numbered, and indexed ADR in one invocation. Use when a decision of real consequence (irreversible, cross-cutting, contested, or convention-setting) has just been made or is being made — including unprompted, when you recognize such a decision mid-task. Invoked by `/os-adr:create`. +description: Record an architecture decision as a correctly templated, numbered, and indexed ADR in one invocation. Use unprompted WHEN a decision of real consequence has just been made or is being made mid-task — choosing between technologies, setting a convention, changing an approach the codebase already uses, or reversing/superseding a previously recorded decision (a reversal is itself a decision that must be recorded). Invoked by `/os-adr:create`. --- Create a new ADR under `docs/adr/`. All mechanical work (ID assignment, templating, index regeneration, supersession bookkeeping) is done by the plugin's CLI — never hand-author the file, number, or index. +**Moments that warrant an ADR** (recognize these unprompted; don't wait to be asked): + +- a technology/library/protocol choice with alternatives that were genuinely considered +- a convention or pattern the codebase will now follow +- a change that replaces an approach the codebase already uses +- a **reversal of an existing Accepted ADR** — a task that reverses a recorded decision is + **not complete** until the superseding ADR is created (`supersedes: NNNN`); implementing + the change and stopping leaves the ADR history asserting the opposite of what the code does + ## 1. Gather the decision content From the conversation (ask only for what you cannot infer): diff --git a/plugins/os-adr/skills/find/SKILL.md b/plugins/os-adr/skills/find/SKILL.md index 067b77f..d21b650 100644 --- a/plugins/os-adr/skills/find/SKILL.md +++ b/plugins/os-adr/skills/find/SKILL.md @@ -1,5 +1,5 @@ --- -description: Retrieve the architecture decision records relevant to files, components, or a topic being worked on — callable at any point mid-task, not just at session start. Use unprompted before changing behavior that may already be decided (a locked choice, a convention, a rejected alternative). Invoked by `/os-adr:find`. +description: Retrieve the architecture decision records relevant to files, components, or a topic being worked on — callable at any point mid-task, not just at session start. Use unprompted WHEN you are about to make an architecture-level choice or change/replace an approach that already exists in the codebase (a persistence choice, a protocol, a library, a convention, an error-handling strategy) — a prior decision may already cover it. Invoked by `/os-adr:find`. --- Find the ADRs relevant to what is being worked on. Retrieval is deterministic-first: run the @@ -27,9 +27,13 @@ matches; treat `graph` matches as "nearby decisions worth a look". ## 3. Act on what you find -- If the in-progress change **conflicts** with an Accepted ADR: stop, surface the specific ADR - (ID + title + the constraint), and either adjust the approach or propose superseding it via - `/os-adr:create` with `supersedes`. +- If an Accepted ADR covers the approach the in-progress change is altering or replacing, you + are **reversing a recorded decision** — and a reversal is itself a decision that must be + recorded, not just implemented. Surface the specific ADR (ID + title + the constraint), and + make the recording offer **explicit and direct** in your reply — state plainly "this reverses + ADR-NNNN; I'll record a superseding ADR via `/os-adr:create`" (or create it outright). Do not + bury the ADR as a parenthetical inside an option list; every path you offer that changes the + decided behavior must carry the explicit superseding-ADR step. - If a candidate merely informs the work, cite it and continue. - If the CLI returns nothing, say so — do not fall back to sweeping `docs/adr/` unless the user asks; an empty result usually means no decision governs those paths (or `affected-paths`