jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/docs/implementation-status/os-adr.md

8.9 KiB
Raw Blame History

os-adr — component detail

Leaf file of ../implementation-status.md. Read on demand.

Global os-adr plugin — cc-os/plugins/os-adr/ (git-tracked, 2026-07-03); symlinked into ~/.claude/plugins/os-adr. Decision records: ADR-0020 (build + open-question resolutions, amended 2026-07-12 for the cc-os retrofit promotion), ADR-0021 (Eval A design).

Component inventory

  • Purpose: Architecture Decision Records, consistently in every project — one customized-Nygard template (frontmatter id/date/status/supersedes/superseded-by/affected-paths/affected-components; Context/Decision/Consequences/Alternatives-rejected sections), docs/adr/NNNN-kebab-title.md one-file-per-decision + fully-regenerated docs/adr/README.md index. Built from docs/adr-system/ (requirements 04, PRD 05, eval sketches 06) via OpenSpec change add-os-adr-plugin.
  • Core: Ruby OO library lib/adr/ (Record, Repository, Index, Template, Detector, Migrator, MigrationReport, Finder; Sandi Metz style, injected paths) behind thin CLIs in bin/ (adr-new, adr-init, adr-detect, adr-migrate, adr-find). Tests: ruby tests/all.rb (47) + python3 tests/hook_test.py (15), all model-free. Behavioral invariants in invariants.md.
  • Hook: moved to os-status (2026-07-06) — the SessionStart existence check now runs as the adr-system-present check in the os-status plugin (hooks.json here is empty; hooks/session_start.py stays as the wording source of record, byte-copied into os-status). Behavior unchanged: present → near-zero-token usage note naming /os-adr:create + /os-adr:find; absent → once-per-day init/migrate suggestion, silenced permanently by .os-adr/suppress (still honored); silent outside git projects.
  • Skills (verb-first, no commands/): create (one-invocation templated+numbered+indexed ADR with mechanical supersession), init, migrate (non-destructive: detect surveyed shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields via --apply-fills → migration_confidence frontmatter + docs/adr/migration-report.md; old-system deletion only as an explicit separate user-approved step), find (deterministic-first retrieval: affected-paths/components match → Accepted-only filter → one-hop Graphify query-path expansion with graceful degradation → AI judgment over the candidate set only).
  • Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25% low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%. Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs need no Graphify tag/edge convention — retrieval layer 3 expands the query paths via graph-node source_files.

Evals

  • Eval A harness (2026-07-03, ADR-0021): plugins/os-adr/eval/ — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (eval/bin/check, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned model:; optimized via the /autoresearch Classic loop over SKILL.md wording only. Procedure: plugins/os-adr/eval/README.md. Grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B — do not conflate.
  • Eval B harness (2026-07-03, OpenSpec change add-os-adr-eval-b-harness): plugins/os-adr/eval-b/ — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) from the frozen shapes in docs/adr-system/06-eval-scenarios.md; Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors); R4's one-hop graph reach uses a real graphify update AST build (rebuilt via eval-b/bin/build-fixture-graph, never committed). Headless-only runner (eval-b/bin/run — fresh claude -p per rep, cwd = sandbox, so the real SessionStart hook fires) and a two-axis deterministic-first checker (eval-b/bin/check): (a) unprompted consultation, mechanical; (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, stubbable via ADR_EVAL_B_JUDGE_CMD). Scenario Task blocks are held-out — never run informally. Procedure: plugins/os-adr/eval-b/README.md. Baseline grid 2026-07-03: haiku 0/8, sonnet 5/8 (fails W3, R1); R4-nograph degradation variant FAILed both tiers as expected. Results: vault note os-adr-eval-b-grid-results-and-observations.
  • Wording experiment complete (2026-07-04): /autoresearch loop over Eval B wording (5 iterations; checker/fixtures/scenarios/rubric frozen) closed the gap — final full grid sonnet 8/8, haiku 7/8 (haiku's one miss is a W3 axis-b judge-boundary flicker). Winning wording shipped in hooks/session_start.py (PRESENT_NOTE), find/create SKILL.mds, and eval-b/fixture/project/CLAUDE.md (the trigger-phrased "Architecture decisions" section — the template for real-project adoption). Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule must live where its precondition is visible (the reversal→supersede rule in the find skill's act-on-findings step fixed W3); lower tiers need mechanical triggers — semantic triggers only reach sonnet. Open: channel ablation never run; R4-nograph now passes both tiers so no longer differentiates. Full mapping: vault note os-adr-eval-b-wording-experiment-hypotheses. Before designing or running any autoresearch eval, Read ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md.
  • Eval B W3 stability check (2026-07-06): 7 headless reps of haiku × W3: axis (a) 7/7 PASS; axis (b) ~5/7 — intermittent flicker, not a hard gap; grid claim stays haiku 7/8; haiku cleared for daily use with that caveat. Recorded in os-adr-eval-b-wording-experiment-hypotheses.
  • Eval C harness (2026-07-06): plugins/os-adr/eval-c/ — ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels (explicit → moderate → conceptual) × paired positive/negative scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications domain). Python taskq fixture with 6-ADR history and the eval-b trigger-phrased CLAUDE.md section. Scoring: positives = both axes; negatives FAIL only on unneeded ADR creation. Model-free bin/self-test green. Scenario Task blocks are held-out — never run informally. Methodology: vault notes eval-methodology-ladder and eval-methodology-irl-feedback-loop; vault gained an eval-results note type + template (2026-07-06).
  • Eval C frozen grid run (2026-07-06): run-set × {sonnet, haiku} × 3 reps/cell (36 headless reps, all counted) — 12/12 cells PASS at ≥2/3 majority; haiku 18/18 reps, sonnet 17/18. Zero over-trigger ADR creation on any negative; every positive pass via governing-ADR citation (16) or judge-recognized proposal (1), never duplicate creation. Sonnet's one FAIL rep is a final-message-citation boundary artifact. Two harness defects fixed on the first live rep, BEFORE the grid (run pipefail bug; mechanical cited-governing PASS branch + two self-test guards). Verdict: the Eval B wording generalizes — grid saturated; next signal is production (rollout + IRL session audits). Full results: vault note os-adr-eval-c-frozen-grid-results.

cc-os retrofit and rollout

  • cc-os ADR retrofit (2026-07-12, via /os-status:fix): the 27-ADR monolithic docs/memory-system/03-architecture-decisions.md was migrated to docs/adr/ with /os-adr:migrate, promoted ahead of the pilot projects (ADR-0020 amended in place). Mechanics: the detector only scans docs/{adr,decisions,decision}/ + DECISIONS.md, so the source was shimmed in via a format-normalized copy under docs/decisions/ (bullet-label → line-start bold-label so the parser could extract sections). Result: 27/27 converted, flag rate 3.7% (one low: 0016), consequences filled per file from source material (sonnet-delegated), 0004/0005/0006 marked Superseded by 0010 with frontmatter links, migration_source repointed at the real monolith path, shim removed. The monolith is retained untouched as the historical source; new decisions go through /os-adr:create.
  • Remaining (rollout order, amended 2026-07-12): real-project migration/adoption one at a time via /os-adr:migrate — pilot projects, then wider. When onboarding real projects, add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from eval-b/fixture/project/CLAUDE.md; candidate: emit it from /os-adr:init/migrate).