diff --git a/docs/plans/2026-07-06-plugin-evals-overview.md b/docs/plans/2026-07-06-plugin-evals-overview.md new file mode 100644 index 0000000..de84a16 --- /dev/null +++ b/docs/plans/2026-07-06-plugin-evals-overview.md @@ -0,0 +1,41 @@ +# Plan set: second-batch plugin evals + status-check convention + +_Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session._ + +Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's +2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared +discipline only. + +## Workstreams + +1. **WS1 — os-orchestration session audit** → `ws1-orchestration-audit.md` + Evidence-gathering over real session transcripts; produces failure-mode clusters and an + eval-scenario backlog. No design dependencies — start anytime. +2. **WS2 — os-vault write-behavior eval** → `ws2-os-vault-write-eval.md` + Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately. + The eval harness itself should start after WS3's design lands (hub-note automation is a + status-check consumer) and after the quick wins (baseline measures post-bugfix behavior). +3. **WS3 — status-check convention + glue plugin** → `ws3-status-convention-plugin.md` + Architecture: OpenSpec change + ADR. Runs in parallel with WS1. + +## Recommended sequencing + +- Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis). +- Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins. +- Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording + loop is needed. + +## Shared discipline (from the first batch — non-negotiable) + +Before designing or running ANY eval below, Read: + +- `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` (run modes, cache + refresh, canary cell, reps, pipefail/rescore gotchas) +- `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, run-set + vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot) +- `~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md` (production audit → new + scenarios) + +Standing rules: held-out Task blocks are never run informally; all reps counted (no +discards); orchestrating session acts as final judge — verify every agent claim against +primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks. diff --git a/docs/plans/ws1-orchestration-audit.md b/docs/plans/ws1-orchestration-audit.md new file mode 100644 index 0000000..72f2366 --- /dev/null +++ b/docs/plans/ws1-orchestration-audit.md @@ -0,0 +1,82 @@ +# WS1 — os-orchestration session audit + +_Created: 2026-07-06. Status: proposed. Prereq: none. See `2026-07-06-plugin-evals-overview.md`._ + +## Goal + +Audit real Claude Code sessions to assess whether delegation under the global +`os-orchestration` plugin is working efficiently, collect concrete miss examples with +triggers/criteria, and distill them into an eval-scenario backlog. This instantiates the +IRL-feedback-loop methodology (production evidence first, scenarios from real misses) — +NOT a lab eval yet. + +## The seven audit questions (fixed rubric — verbatim from the user) + +1. Are subagents getting called when they should be? +2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost? +3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use? +4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would + read anyway)? +5. Is the orchestrator sharing too much context with subagents (filling their windows / + clouding judgment)? +6. Is the orchestrator even following the ORCHESTRATION.md instructions? +7. Is the orchestrator requesting/receiving back only the context it needs, rather than a + full context dump from the subagent? + +## Phase 1 — deterministic extractor (orchestrator directly; one script) + +Raw transcripts are multi-MB jsonl; feeding them whole to auditors violates the principle +being audited. Write one script (Python or Ruby; Ruby preferred per user style if it stays +clean) that reduces a session jsonl to a delegation fact-sheet: + +- Every `Agent` tool_use: agent type, `model` param present? which?, prompt char count, + run_in_background, result char count returned to orchestrator. +- Orchestrator's own tool profile: Read/Grep/Glob/Bash counts and total bytes read, + bucketed before/after each Agent spawn. +- Candidate missed delegations: runs of ≥N same-type tool calls across independent files + with no Agent spawn (mechanical heuristic; auditor judges). +- Session metadata: project dir, date, total turns, model of main loop if recoverable. + +Transcripts live under `~/.claude/projects//*.jsonl`. +This extractor is dual-use: it becomes the checker core for any later orchestration eval. + +## Phase 2 — sample selection (orchestrator, from fact-sheets not vibes) + +~8–10 sessions, stratified: +- 3 × cc-os (`-home-jared-dev-cc-os`, 70 sessions available) — heavy orchestration use, + but note contamination: these sessions were actively working ON the orchestration text. +- 3 × client/dev projects (`philly-search-engine-marketing` 28, `llf-schema` 26, + `viking-warrior-training-log` 18). +- 2 × ops (`-home-jared-servers` 19, `-home-jared-systems-admin` 17). +- Bias recent (post-2026-07-03, when os-orchestration went global — earlier sessions ran + under different/per-project orchestration text and are a different population). +- Include sessions WITH delegation and large sessions with NONE despite apparent + parallelizable work. + +## Phase 3 — parallel audits (sonnet subagents, one per session) + +Each auditor receives: the fact-sheet, the transcript path (for targeted spot-reads only — +instruct them NOT to read the whole file), the ORCHESTRATION.md text (10 lines), and the +seven questions. Required output per question: verdict + cited evidence (jsonl line refs) + +if a miss, a candidate eval trigger/criterion ("when session state X, orchestrator should +Y"). Model: sonnet (judgment work). Spawn all auditors concurrently. + +## Phase 4 — synthesis (orchestrator as final judge) + +Verify each claimed miss against the primary transcript before accepting (standing rule — +auditor reports have contradicted primary evidence before). Cluster confirmed misses into +failure modes; each cluster → an eval-scenario candidate with a measurable trigger. + +Deliverables: +- Findings doc: `docs/orchestration-audit/2026-07-XX-findings.md` (per-question summary, + per-session evidence table, failure-mode clusters). +- Scenario backlog appended to the findings doc — inputs to a future eval design, NOT an + eval yet. + +## Eval-design note (deferred, do not start in this workstream) + +The Eval B headless pattern does not transfer directly: orchestration behavior is +mid-session and depends on task shape, not a SessionStart trigger. Likely shape: scripted +multi-file tasks run headless, scored by the Phase 1 extractor + thresholds (did it +delegate? model passed? orchestrator bytes-read under budget?). Design only after Phase 4 +findings exist — scenarios must come from real misses, not imagination. diff --git a/docs/plans/ws2-os-vault-write-eval.md b/docs/plans/ws2-os-vault-write-eval.md new file mode 100644 index 0000000..37c90f6 --- /dev/null +++ b/docs/plans/ws2-os-vault-write-eval.md @@ -0,0 +1,94 @@ +# WS2 — os-vault "when and how to write" eval + +_Created: 2026-07-06. Status: proposed. Prereqs: quick wins immediate; harness after WS3 +design + quick wins. See `2026-07-06-plugin-evals-overview.md`._ + +## Goal + +Make os-vault reliably know WHEN to write a vault note unprompted and HOW to write it +correctly (frontmatter contract), measured by a held-out discrimination eval — applying the +full os-adr first-batch playbook. + +## Known defects found 2026-07-06 (fix before baselining) + +1. **`skills/write/SKILL.md` contradicts the reconciled vault-conventions.md.** The skill + still mandates `scope/global` as a TAG and a type list of + `procedure|reference|log|hub|concept|decision`; vault-conventions (Phase 1 SB content + plan, 2026-06-30) made `scope:` a frontmatter FIELD and the live vault uses types like + `howto` and `eval-results`. Fix: make vault-conventions.md the single source of truth — + the skill summarizes + points, never duplicates the schema. Then `bin/refresh-plugins`. +2. **No trigger-conditioned wording anywhere in os-vault.** The core Eval B lesson + (when→then phrasing; mechanical triggers for lower tiers; each rule lives where its + precondition is visible) was never applied. Nothing tells the model when to invoke + `/os-vault:write` unprompted. Do NOT fix this before the baseline grid — it is the + tuning surface the eval measures. +3. **cc-os has no project hub note** despite hub notes being the connectivity mechanism + (ADR-014). Quick win: author it now (one note). AUTOMATIC hub creation is designed into + `onboard-project` + the WS3 status check — not hacked in here. + +Quick wins = items 1 and 3. Both are small; do directly or via one haiku subagent each. + +## Behavior under test + +Unprompted vault-write discrimination during a normal working session: + +- **Positives:** session surfaces evergreen cross-project knowledge (tool/API behavior + discovered, client-specific fact, a convention established with the user) → model writes + or offers a vault note with contract-conforming frontmatter (summary at creation, scope + field, type/ + ≥1 facet tag). +- **Negatives (equally tempting):** ephemeral material — in-progress task state, + project-local debugging detail, decisions already recorded in repo docs/ADRs → no vault + write. Over-trigger fail line: writing ephemera to the vault (or duplicating + repo-recorded knowledge). + +Ambiguity ladder: L1 explicit cue ("worth remembering") → L2 moderate → L3 conceptual +(knowledge emerges implicitly from debugging). Paired positive/negative per level. + +## Harness (Eval B/C shape) + +- Location: `plugins/os-vault/eval/`. Run-set + frozen reserve authored upfront (2×), in + different knowledge domains. **Task blocks held-out — never run informally; reserve + never read informally.** +- **Headless-only** (`claude -p`, cwd = sandbox fixture project, real SessionStart hook + fires). In-session subagents are invalid for unprompted-behavior measurement. +- Sandbox must use an ISOLATED vault path (env-pointed test vault), never the real + `~/Documents/SecondBrain` — the checker inspects the sandbox vault for written notes. + Verify the plugin's vault path is env-overridable (config.yaml); if not, that's a small + pre-harness plugin change. +- Checker: Ruby (Sandi Metz style), deterministic-first. Axis (a): did it write/offer a + vault note (mechanical from transcript tool_use + sandbox vault diff). Axis (b): + frontmatter contract conformance — almost fully mechanical (summary present, scope + field, required tag groups); narrow frozen haiku judge only for "offered but did not + write" phrasing, stubbable via env var. Negatives: sole fail line = unneeded vault + write; consultation informational. TSV mode for grids. + +## Discipline gates (all mandatory before the first grid) + +1. Conformance dry-run on paper: would a perfectly write-skill-compliant model pass every + cell? Would an always-write model pass positives but fail negatives? +2. Model-free self-test including at least one fabricated SHIPPED-INSTRUCTION-COMPLIANT + transcript (not just the designer's imagined ideal). +3. Canary cell first (one cell, hand-verify TSV vs raw transcript, count the result). +4. `bin/refresh-plugins` before every grid; check TSV row counts == expected reps + (pipefail/tee gotcha); rescore-don't-discard on instrument fixes. + +## Sequence + +1. Quick wins (contract fix + hub note) → refresh caches. +2. Build harness (fixture project + isolated vault + 6-ADR-equivalent seeded vault history + for negatives to cite): fixture generation → sonnet subagent; checker → sonnet subagent + (Ruby); scenario authoring + conformance dry-run → ORCHESTRATOR DIRECTLY (Eval C's + defects came from exactly this step; do not delegate it). +3. **Baseline grid untuned** (post-bugfix, pre-trigger-wording): run-set × {sonnet, haiku} + × 3 reps. This measures the actual gap. +4. If gapped: `/autoresearch` wording loop per the howto note (checker/fixtures/scenarios + frozen; only wording moves; reduced inner grid; full grid to confirm). Expected surfaces: + session_start hook note, write SKILL.md description + body, possibly a CLAUDE.md + trigger-phrased section template (mirror os-adr's). +5. Results → vault `eval-results` note (template `_templates/eval-results.md`); update + cc-os CLAUDE.md pointers. + +## Models + +Sonnet for fixture/checker construction and any judging; haiku for mechanical edits; +orchestrator authors scenarios and acts as final judge over all grid claims. diff --git a/docs/plans/ws3-status-convention-plugin.md b/docs/plans/ws3-status-convention-plugin.md new file mode 100644 index 0000000..3d39909 --- /dev/null +++ b/docs/plans/ws3-status-convention-plugin.md @@ -0,0 +1,91 @@ +# WS3 — per-plugin SessionStart status-check convention + glue plugin + +_Created: 2026-07-06. Status: proposed. Prereq: none (parallel with WS1). See +`2026-07-06-plugin-evals-overview.md`._ + +## Goal + +A convention-over-configuration status system: at SessionStart, every installed cc-os +plugin verifies its expected per-project artifacts exist (adr: ADR system present; vault: +project hub note exists — optionally injecting it as context). All good → silent. Problem → +one short banner prompting corrective action (" missing — run /os-vault:… to fix"). +Plugins stay independent; new plugins get picked up automatically by following the +convention; only the glue plugin is mandatory for the collection. + +This is architecture → **OpenSpec change + ADR** (use `openspec-propose`; decisions land in +`docs/memory-system/03-architecture-decisions.md`). Do not build outside that workflow. + +## User's sketch (the starting design) + +Per-plugin Python status scripts following a cc-os convention (naming, interface, +location), living in a specific gitignored hidden directory in the project. SessionStart +fires a cc-os master status script that enumerates all status scripts in that directory +(excluding itself) and runs them. Adding a plugin later = drop in a conforming script; no +global rewrites. + +## The one design tension the proposal must settle (pressure-test, don't fiat) + +**Where does status-check CODE live?** + +- **Option A (user sketch): project-local dir** (e.g. `.cc-os/status.d/*.py`, gitignored). + Pro: per-project opt-in/customization. Con: code copies drift across projects; needs an + install/update step per project per plugin; the 2026-07-04 stale-cache incident shows + copy-drift is a real failure class here. +- **Option B: plugin-source enumeration** — each plugin ships `status/check.py` in its own + source; the master enumerates installed `os-*` plugins at session start. Same modularity + and zero-global-rewrite extension; no copies, no drift, no install step. Project-local + `.cc-os/` still exists but holds STATE only (suppress files, per-project config like + "this project opts out of hub-note checks"), mirroring os-adr's `.os-adr/` and + os-doc-hygiene's `.dochygiene/` — consolidating those is itself a candidate scope item. + +Run `perspectives:simplifier` and `perspectives:implementer` (both sonnet) against both +shapes inside the proposal. Lean: B for code, A-style dir for state. + +## Interface contract (draft for the proposal) + +- Each check: stdin/argv gets project root + plugin config; stdout JSON + `{"status": "ok"|"warn", "message": str, "context": str|null}`; hard timeout (~5s); + non-zero exit or timeout = treated as warn with a generic message, never blocks session. +- **Silent on ok** — os-adr's near-zero-token SessionStart discipline is the template. + Master aggregates warns into ONE short banner; per-project once-per-day snooze + + permanent suppress via state dir (pattern already proven in os-adr/os-doc-hygiene). +- `context` field = optional session-context injection (the "pull the hub note into + context if useful" idea) — but gate it through injection-economics thinking (see the + design-template skill's filter); default null. + +## Glue plugin scope ("cc-os collection manager") + +Naming: per `~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` (read before +naming; `os-[domain]`, verb-first skills, NO `name:` frontmatter). Candidates: `os-core` +(lean) or `os-status` (narrower). Settle in the proposal. + +Phase 1 (this change): master SessionStart hook + enumeration + interface contract + +convert the two existing consumers: +- os-adr: refactor its existing existence-check hook into a conforming status check (it is + already exactly this pattern — present→usage note, absent→once-per-day suggestion). +- os-vault: new hub-note check (project hub note exists for the current project; missing → + prompt to run onboarding/creation; optionally inject on present). + +Phase 2 (separate change, later): collection-manager duties — verify installed cc-os +plugin caches are fresh vs source (would have auto-caught the 2026-07-04 stale-cache +incident), migrations complete, conventions followed (e.g. no `name:` in SKILL.md +frontmatter). Effectively an automated `bin/refresh-plugins` advisor. + +## Deliverables + +1. OpenSpec change (design.md resolving the tension above + specs + tasks) — via + `openspec-propose`. +2. ADR in `docs/memory-system/03-architecture-decisions.md` (status-check convention + + code-location decision). +3. Implementation via `openspec-apply-change`: master hook + shared contract module + (Python, matching os-vault's deep-module style: thin entry points, shared config/io + modules), conforming checks for os-adr + os-vault, model-free tests (Python, mirroring + `tests/hook_test.py` style) + invariants.md. +4. `bin/refresh-plugins`, `claude plugin details` verification, fresh-session smoke test. + +## Models + +Design synthesis + ADR: orchestrator. Perspective reviews: sonnet. Implementation: haiku +for thin scripts/tests, sonnet for the shared contract module. No behavioral eval needed — +deterministic code; invariants + model-free tests suffice (same rationale as the os-adr +hook).