122 lines
8.9 KiB
Markdown
122 lines
8.9 KiB
Markdown
# os-adr — component detail
|
||
|
||
_Leaf file of [../implementation-status.md](../implementation-status.md). Read on demand._
|
||
|
||
**Global os-adr plugin** — `cc-os/plugins/os-adr/` (git-tracked, 2026-07-03); symlinked into
|
||
`~/.claude/plugins/os-adr`. Decision records: ADR-0020 (build + open-question resolutions,
|
||
amended 2026-07-12 for the cc-os retrofit promotion), ADR-0021 (Eval A design).
|
||
|
||
## Component inventory
|
||
|
||
- Purpose: Architecture Decision Records, consistently in every project — one
|
||
customized-Nygard template (frontmatter
|
||
`id`/`date`/`status`/`supersedes`/`superseded-by`/`affected-paths`/`affected-components`;
|
||
Context/Decision/Consequences/Alternatives-rejected sections),
|
||
`docs/adr/NNNN-kebab-title.md` one-file-per-decision + fully-regenerated
|
||
`docs/adr/README.md` index. Built from `docs/adr-system/` (requirements `04`, PRD `05`,
|
||
eval sketches `06`) via OpenSpec change `add-os-adr-plugin`.
|
||
- Core: Ruby OO library `lib/adr/` (`Record`, `Repository`, `Index`, `Template`, `Detector`,
|
||
`Migrator`, `MigrationReport`, `Finder`; Sandi Metz style, injected paths) behind thin
|
||
CLIs in `bin/` (`adr-new`, `adr-init`, `adr-detect`, `adr-migrate`, `adr-find`). Tests:
|
||
`ruby tests/all.rb` (47) + `python3 tests/hook_test.py` (15), all model-free. Behavioral
|
||
invariants in `invariants.md`.
|
||
- Hook: **moved to os-status (2026-07-06)** — the SessionStart existence check now runs as
|
||
the `adr-system-present` check in the os-status plugin (`hooks.json` here is empty;
|
||
`hooks/session_start.py` stays as the wording source of record, byte-copied into
|
||
os-status). Behavior unchanged: present → near-zero-token usage note naming
|
||
`/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced
|
||
permanently by `.os-adr/suppress` (still honored); silent outside git projects.
|
||
- Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed
|
||
ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed
|
||
shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields
|
||
via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`;
|
||
old-system deletion only as an explicit separate user-approved step), `find`
|
||
(deterministic-first retrieval: affected-paths/components match → Accepted-only filter →
|
||
one-hop Graphify query-path expansion with graceful degradation → AI judgment over the
|
||
candidate set only).
|
||
- Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25%
|
||
low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%.
|
||
Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs
|
||
need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via
|
||
graph-node `source_file`s.
|
||
|
||
## Evals
|
||
|
||
- **Eval A harness (2026-07-03, ADR-0021):** `plugins/os-adr/eval/` — prompted
|
||
skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the
|
||
plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4
|
||
find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`,
|
||
structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts,
|
||
runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned
|
||
`model:`; optimized via the `/autoresearch` Classic loop over SKILL.md *wording only*.
|
||
Procedure: `plugins/os-adr/eval/README.md`. Grid run 2026-07-03, all scenarios × both
|
||
tiers passing. Distinct from the held-out Eval B — do not conflate.
|
||
- **Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`):**
|
||
`plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7
|
||
scenarios (W1–W3 write-trigger, R1–R4 retrieval) from the frozen shapes in
|
||
`docs/adr-system/06-eval-scenarios.md`; Ruby webhook-relay fixture with a 6-ADR history
|
||
(Superseded pair + near-miss distractors); R4's one-hop graph reach uses a real
|
||
`graphify update` AST build (rebuilt via `eval-b/bin/build-fixture-graph`, never
|
||
committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd =
|
||
sandbox, so the real SessionStart hook fires) and a two-axis deterministic-first checker
|
||
(`eval-b/bin/check`): (a) unprompted consultation, mechanical; (b) correct-ADR citation
|
||
(R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3,
|
||
stubbable via `ADR_EVAL_B_JUDGE_CMD`). **Scenario Task blocks are held-out — never run
|
||
informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Baseline grid 2026-07-03:
|
||
haiku 0/8, sonnet 5/8 (fails W3, R1); `R4-nograph` degradation variant FAILed both tiers
|
||
as expected. Results: vault note [[os-adr-eval-b-grid-results-and-observations]].
|
||
- **Wording experiment complete (2026-07-04):** `/autoresearch` loop over Eval B wording (5
|
||
iterations; checker/fixtures/scenarios/rubric frozen) closed the gap — final full grid
|
||
**sonnet 8/8, haiku 7/8** (haiku's one miss is a W3 axis-b judge-boundary flicker).
|
||
Winning wording shipped in `hooks/session_start.py` (PRESENT_NOTE), find/create SKILL.mds,
|
||
and `eval-b/fixture/project/CLAUDE.md` (the trigger-phrased "Architecture decisions"
|
||
section — the template for real-project adoption). Confirmed mechanisms:
|
||
trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule
|
||
must live where its precondition is visible (the reversal→supersede rule in the find
|
||
skill's act-on-findings step fixed W3); lower tiers need *mechanical* triggers — semantic
|
||
triggers only reach sonnet. Open: channel ablation never run; R4-nograph now passes both
|
||
tiers so no longer differentiates. Full mapping: vault note
|
||
[[os-adr-eval-b-wording-experiment-hypotheses]]. **Before designing or running any
|
||
autoresearch eval, Read
|
||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.**
|
||
- **Eval B W3 stability check (2026-07-06):** 7 headless reps of haiku × W3: axis (a) 7/7
|
||
PASS; axis (b) ~5/7 — intermittent flicker, not a hard gap; grid claim stays **haiku
|
||
7/8**; haiku cleared for daily use with that caveat. Recorded in
|
||
[[os-adr-eval-b-wording-experiment-hypotheses]].
|
||
- **Eval C harness (2026-07-06):** `plugins/os-adr/eval-c/` — ambiguity-ladder
|
||
DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels
|
||
(explicit → moderate → conceptual) × paired positive/negative scenarios; run-set (6,
|
||
job-execution domain) + frozen reserve-set (6, notifications domain). Python `taskq`
|
||
fixture with 6-ADR history and the eval-b trigger-phrased CLAUDE.md section. Scoring:
|
||
positives = both axes; negatives FAIL **only on unneeded ADR creation**. Model-free
|
||
`bin/self-test` green. **Scenario Task blocks are held-out — never run informally.**
|
||
Methodology: vault notes [[eval-methodology-ladder]] and
|
||
[[eval-methodology-irl-feedback-loop]]; vault gained an `eval-results` note type +
|
||
template (2026-07-06).
|
||
- **Eval C frozen grid run (2026-07-06):** run-set × {sonnet, haiku} × 3 reps/cell (36
|
||
headless reps, all counted) — **12/12 cells PASS at ≥2/3 majority; haiku 18/18 reps,
|
||
sonnet 17/18**. Zero over-trigger ADR creation on any negative; every positive pass via
|
||
governing-ADR citation (16) or judge-recognized proposal (1), never duplicate creation.
|
||
Sonnet's one FAIL rep is a final-message-citation boundary artifact. Two harness defects
|
||
fixed on the first live rep, BEFORE the grid (run pipefail bug; mechanical
|
||
`cited-governing` PASS branch + two self-test guards). Verdict: the Eval B wording
|
||
**generalizes** — grid saturated; next signal is production (rollout + IRL session
|
||
audits). Full results: vault note [[os-adr-eval-c-frozen-grid-results]].
|
||
|
||
## cc-os retrofit and rollout
|
||
|
||
- **cc-os ADR retrofit (2026-07-12, via `/os-status:fix`):** the 27-ADR monolithic
|
||
`docs/memory-system/03-architecture-decisions.md` was migrated to `docs/adr/` with
|
||
`/os-adr:migrate`, promoted ahead of the pilot projects (ADR-0020 amended in place).
|
||
Mechanics: the detector only scans `docs/{adr,decisions,decision}/` + `DECISIONS.md`, so
|
||
the source was shimmed in via a format-normalized copy under `docs/decisions/`
|
||
(bullet-label → line-start bold-label so the parser could extract sections). Result: 27/27
|
||
converted, flag rate 3.7% (one low: 0016), consequences filled per file from source
|
||
material (sonnet-delegated), 0004/0005/0006 marked Superseded by 0010 with frontmatter
|
||
links, `migration_source` repointed at the real monolith path, shim removed. The monolith
|
||
is retained untouched as the historical source; new decisions go through `/os-adr:create`.
|
||
- **Remaining (rollout order, amended 2026-07-12):** real-project migration/adoption one at
|
||
a time via `/os-adr:migrate` — pilot projects, then wider. When onboarding real projects,
|
||
add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from
|
||
`eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`).
|