Add second-batch plugin eval and architecture plans
Three workstreams with shared methodology: os-orchestration session audit (WS1), os-vault write-behavior eval (WS2), and status-check convention + glue plugin (WS3). Each specifies phases, discipline gates, and deliverables, following the audit-first and eval-B/C playbook established in the first batch. To be triggered in separate future sessions.
This commit is contained in:
jared
parent
2e9d83f749
commit
973fbe7a66
|
|
@ -0,0 +1,41 @@
|
||||||
|
# Plan set: second-batch plugin evals + status-check convention
|
||||||
|
|
||||||
|
_Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session._
|
||||||
|
|
||||||
|
Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's
|
||||||
|
2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared
|
||||||
|
discipline only.
|
||||||
|
|
||||||
|
## Workstreams
|
||||||
|
|
||||||
|
1. **WS1 — os-orchestration session audit** → `ws1-orchestration-audit.md`
|
||||||
|
Evidence-gathering over real session transcripts; produces failure-mode clusters and an
|
||||||
|
eval-scenario backlog. No design dependencies — start anytime.
|
||||||
|
2. **WS2 — os-vault write-behavior eval** → `ws2-os-vault-write-eval.md`
|
||||||
|
Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately.
|
||||||
|
The eval harness itself should start after WS3's design lands (hub-note automation is a
|
||||||
|
status-check consumer) and after the quick wins (baseline measures post-bugfix behavior).
|
||||||
|
3. **WS3 — status-check convention + glue plugin** → `ws3-status-convention-plugin.md`
|
||||||
|
Architecture: OpenSpec change + ADR. Runs in parallel with WS1.
|
||||||
|
|
||||||
|
## Recommended sequencing
|
||||||
|
|
||||||
|
- Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis).
|
||||||
|
- Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins.
|
||||||
|
- Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording
|
||||||
|
loop is needed.
|
||||||
|
|
||||||
|
## Shared discipline (from the first batch — non-negotiable)
|
||||||
|
|
||||||
|
Before designing or running ANY eval below, Read:
|
||||||
|
|
||||||
|
- `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` (run modes, cache
|
||||||
|
refresh, canary cell, reps, pipefail/rescore gotchas)
|
||||||
|
- `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, run-set
|
||||||
|
vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot)
|
||||||
|
- `~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md` (production audit → new
|
||||||
|
scenarios)
|
||||||
|
|
||||||
|
Standing rules: held-out Task blocks are never run informally; all reps counted (no
|
||||||
|
discards); orchestrating session acts as final judge — verify every agent claim against
|
||||||
|
primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks.
|
||||||
|
|
@ -0,0 +1,82 @@
|
||||||
|
# WS1 — os-orchestration session audit
|
||||||
|
|
||||||
|
_Created: 2026-07-06. Status: proposed. Prereq: none. See `2026-07-06-plugin-evals-overview.md`._
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Audit real Claude Code sessions to assess whether delegation under the global
|
||||||
|
`os-orchestration` plugin is working efficiently, collect concrete miss examples with
|
||||||
|
triggers/criteria, and distill them into an eval-scenario backlog. This instantiates the
|
||||||
|
IRL-feedback-loop methodology (production evidence first, scenarios from real misses) —
|
||||||
|
NOT a lab eval yet.
|
||||||
|
|
||||||
|
## The seven audit questions (fixed rubric — verbatim from the user)
|
||||||
|
|
||||||
|
1. Are subagents getting called when they should be?
|
||||||
|
2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
|
||||||
|
3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
|
||||||
|
4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would
|
||||||
|
read anyway)?
|
||||||
|
5. Is the orchestrator sharing too much context with subagents (filling their windows /
|
||||||
|
clouding judgment)?
|
||||||
|
6. Is the orchestrator even following the ORCHESTRATION.md instructions?
|
||||||
|
7. Is the orchestrator requesting/receiving back only the context it needs, rather than a
|
||||||
|
full context dump from the subagent?
|
||||||
|
|
||||||
|
## Phase 1 — deterministic extractor (orchestrator directly; one script)
|
||||||
|
|
||||||
|
Raw transcripts are multi-MB jsonl; feeding them whole to auditors violates the principle
|
||||||
|
being audited. Write one script (Python or Ruby; Ruby preferred per user style if it stays
|
||||||
|
clean) that reduces a session jsonl to a delegation fact-sheet:
|
||||||
|
|
||||||
|
- Every `Agent` tool_use: agent type, `model` param present? which?, prompt char count,
|
||||||
|
run_in_background, result char count returned to orchestrator.
|
||||||
|
- Orchestrator's own tool profile: Read/Grep/Glob/Bash counts and total bytes read,
|
||||||
|
bucketed before/after each Agent spawn.
|
||||||
|
- Candidate missed delegations: runs of ≥N same-type tool calls across independent files
|
||||||
|
with no Agent spawn (mechanical heuristic; auditor judges).
|
||||||
|
- Session metadata: project dir, date, total turns, model of main loop if recoverable.
|
||||||
|
|
||||||
|
Transcripts live under `~/.claude/projects/<flattened-path>/*.jsonl`.
|
||||||
|
This extractor is dual-use: it becomes the checker core for any later orchestration eval.
|
||||||
|
|
||||||
|
## Phase 2 — sample selection (orchestrator, from fact-sheets not vibes)
|
||||||
|
|
||||||
|
~8–10 sessions, stratified:
|
||||||
|
- 3 × cc-os (`-home-jared-dev-cc-os`, 70 sessions available) — heavy orchestration use,
|
||||||
|
but note contamination: these sessions were actively working ON the orchestration text.
|
||||||
|
- 3 × client/dev projects (`philly-search-engine-marketing` 28, `llf-schema` 26,
|
||||||
|
`viking-warrior-training-log` 18).
|
||||||
|
- 2 × ops (`-home-jared-servers` 19, `-home-jared-systems-admin` 17).
|
||||||
|
- Bias recent (post-2026-07-03, when os-orchestration went global — earlier sessions ran
|
||||||
|
under different/per-project orchestration text and are a different population).
|
||||||
|
- Include sessions WITH delegation and large sessions with NONE despite apparent
|
||||||
|
parallelizable work.
|
||||||
|
|
||||||
|
## Phase 3 — parallel audits (sonnet subagents, one per session)
|
||||||
|
|
||||||
|
Each auditor receives: the fact-sheet, the transcript path (for targeted spot-reads only —
|
||||||
|
instruct them NOT to read the whole file), the ORCHESTRATION.md text (10 lines), and the
|
||||||
|
seven questions. Required output per question: verdict + cited evidence (jsonl line refs) +
|
||||||
|
if a miss, a candidate eval trigger/criterion ("when session state X, orchestrator should
|
||||||
|
Y"). Model: sonnet (judgment work). Spawn all auditors concurrently.
|
||||||
|
|
||||||
|
## Phase 4 — synthesis (orchestrator as final judge)
|
||||||
|
|
||||||
|
Verify each claimed miss against the primary transcript before accepting (standing rule —
|
||||||
|
auditor reports have contradicted primary evidence before). Cluster confirmed misses into
|
||||||
|
failure modes; each cluster → an eval-scenario candidate with a measurable trigger.
|
||||||
|
|
||||||
|
Deliverables:
|
||||||
|
- Findings doc: `docs/orchestration-audit/2026-07-XX-findings.md` (per-question summary,
|
||||||
|
per-session evidence table, failure-mode clusters).
|
||||||
|
- Scenario backlog appended to the findings doc — inputs to a future eval design, NOT an
|
||||||
|
eval yet.
|
||||||
|
|
||||||
|
## Eval-design note (deferred, do not start in this workstream)
|
||||||
|
|
||||||
|
The Eval B headless pattern does not transfer directly: orchestration behavior is
|
||||||
|
mid-session and depends on task shape, not a SessionStart trigger. Likely shape: scripted
|
||||||
|
multi-file tasks run headless, scored by the Phase 1 extractor + thresholds (did it
|
||||||
|
delegate? model passed? orchestrator bytes-read under budget?). Design only after Phase 4
|
||||||
|
findings exist — scenarios must come from real misses, not imagination.
|
||||||
|
|
@ -0,0 +1,94 @@
|
||||||
|
# WS2 — os-vault "when and how to write" eval
|
||||||
|
|
||||||
|
_Created: 2026-07-06. Status: proposed. Prereqs: quick wins immediate; harness after WS3
|
||||||
|
design + quick wins. See `2026-07-06-plugin-evals-overview.md`._
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Make os-vault reliably know WHEN to write a vault note unprompted and HOW to write it
|
||||||
|
correctly (frontmatter contract), measured by a held-out discrimination eval — applying the
|
||||||
|
full os-adr first-batch playbook.
|
||||||
|
|
||||||
|
## Known defects found 2026-07-06 (fix before baselining)
|
||||||
|
|
||||||
|
1. **`skills/write/SKILL.md` contradicts the reconciled vault-conventions.md.** The skill
|
||||||
|
still mandates `scope/global` as a TAG and a type list of
|
||||||
|
`procedure|reference|log|hub|concept|decision`; vault-conventions (Phase 1 SB content
|
||||||
|
plan, 2026-06-30) made `scope:` a frontmatter FIELD and the live vault uses types like
|
||||||
|
`howto` and `eval-results`. Fix: make vault-conventions.md the single source of truth —
|
||||||
|
the skill summarizes + points, never duplicates the schema. Then `bin/refresh-plugins`.
|
||||||
|
2. **No trigger-conditioned wording anywhere in os-vault.** The core Eval B lesson
|
||||||
|
(when→then phrasing; mechanical triggers for lower tiers; each rule lives where its
|
||||||
|
precondition is visible) was never applied. Nothing tells the model when to invoke
|
||||||
|
`/os-vault:write` unprompted. Do NOT fix this before the baseline grid — it is the
|
||||||
|
tuning surface the eval measures.
|
||||||
|
3. **cc-os has no project hub note** despite hub notes being the connectivity mechanism
|
||||||
|
(ADR-014). Quick win: author it now (one note). AUTOMATIC hub creation is designed into
|
||||||
|
`onboard-project` + the WS3 status check — not hacked in here.
|
||||||
|
|
||||||
|
Quick wins = items 1 and 3. Both are small; do directly or via one haiku subagent each.
|
||||||
|
|
||||||
|
## Behavior under test
|
||||||
|
|
||||||
|
Unprompted vault-write discrimination during a normal working session:
|
||||||
|
|
||||||
|
- **Positives:** session surfaces evergreen cross-project knowledge (tool/API behavior
|
||||||
|
discovered, client-specific fact, a convention established with the user) → model writes
|
||||||
|
or offers a vault note with contract-conforming frontmatter (summary at creation, scope
|
||||||
|
field, type/ + ≥1 facet tag).
|
||||||
|
- **Negatives (equally tempting):** ephemeral material — in-progress task state,
|
||||||
|
project-local debugging detail, decisions already recorded in repo docs/ADRs → no vault
|
||||||
|
write. Over-trigger fail line: writing ephemera to the vault (or duplicating
|
||||||
|
repo-recorded knowledge).
|
||||||
|
|
||||||
|
Ambiguity ladder: L1 explicit cue ("worth remembering") → L2 moderate → L3 conceptual
|
||||||
|
(knowledge emerges implicitly from debugging). Paired positive/negative per level.
|
||||||
|
|
||||||
|
## Harness (Eval B/C shape)
|
||||||
|
|
||||||
|
- Location: `plugins/os-vault/eval/`. Run-set + frozen reserve authored upfront (2×), in
|
||||||
|
different knowledge domains. **Task blocks held-out — never run informally; reserve
|
||||||
|
never read informally.**
|
||||||
|
- **Headless-only** (`claude -p`, cwd = sandbox fixture project, real SessionStart hook
|
||||||
|
fires). In-session subagents are invalid for unprompted-behavior measurement.
|
||||||
|
- Sandbox must use an ISOLATED vault path (env-pointed test vault), never the real
|
||||||
|
`~/Documents/SecondBrain` — the checker inspects the sandbox vault for written notes.
|
||||||
|
Verify the plugin's vault path is env-overridable (config.yaml); if not, that's a small
|
||||||
|
pre-harness plugin change.
|
||||||
|
- Checker: Ruby (Sandi Metz style), deterministic-first. Axis (a): did it write/offer a
|
||||||
|
vault note (mechanical from transcript tool_use + sandbox vault diff). Axis (b):
|
||||||
|
frontmatter contract conformance — almost fully mechanical (summary present, scope
|
||||||
|
field, required tag groups); narrow frozen haiku judge only for "offered but did not
|
||||||
|
write" phrasing, stubbable via env var. Negatives: sole fail line = unneeded vault
|
||||||
|
write; consultation informational. TSV mode for grids.
|
||||||
|
|
||||||
|
## Discipline gates (all mandatory before the first grid)
|
||||||
|
|
||||||
|
1. Conformance dry-run on paper: would a perfectly write-skill-compliant model pass every
|
||||||
|
cell? Would an always-write model pass positives but fail negatives?
|
||||||
|
2. Model-free self-test including at least one fabricated SHIPPED-INSTRUCTION-COMPLIANT
|
||||||
|
transcript (not just the designer's imagined ideal).
|
||||||
|
3. Canary cell first (one cell, hand-verify TSV vs raw transcript, count the result).
|
||||||
|
4. `bin/refresh-plugins` before every grid; check TSV row counts == expected reps
|
||||||
|
(pipefail/tee gotcha); rescore-don't-discard on instrument fixes.
|
||||||
|
|
||||||
|
## Sequence
|
||||||
|
|
||||||
|
1. Quick wins (contract fix + hub note) → refresh caches.
|
||||||
|
2. Build harness (fixture project + isolated vault + 6-ADR-equivalent seeded vault history
|
||||||
|
for negatives to cite): fixture generation → sonnet subagent; checker → sonnet subagent
|
||||||
|
(Ruby); scenario authoring + conformance dry-run → ORCHESTRATOR DIRECTLY (Eval C's
|
||||||
|
defects came from exactly this step; do not delegate it).
|
||||||
|
3. **Baseline grid untuned** (post-bugfix, pre-trigger-wording): run-set × {sonnet, haiku}
|
||||||
|
× 3 reps. This measures the actual gap.
|
||||||
|
4. If gapped: `/autoresearch` wording loop per the howto note (checker/fixtures/scenarios
|
||||||
|
frozen; only wording moves; reduced inner grid; full grid to confirm). Expected surfaces:
|
||||||
|
session_start hook note, write SKILL.md description + body, possibly a CLAUDE.md
|
||||||
|
trigger-phrased section template (mirror os-adr's).
|
||||||
|
5. Results → vault `eval-results` note (template `_templates/eval-results.md`); update
|
||||||
|
cc-os CLAUDE.md pointers.
|
||||||
|
|
||||||
|
## Models
|
||||||
|
|
||||||
|
Sonnet for fixture/checker construction and any judging; haiku for mechanical edits;
|
||||||
|
orchestrator authors scenarios and acts as final judge over all grid claims.
|
||||||
|
|
@ -0,0 +1,91 @@
|
||||||
|
# WS3 — per-plugin SessionStart status-check convention + glue plugin
|
||||||
|
|
||||||
|
_Created: 2026-07-06. Status: proposed. Prereq: none (parallel with WS1). See
|
||||||
|
`2026-07-06-plugin-evals-overview.md`._
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
A convention-over-configuration status system: at SessionStart, every installed cc-os
|
||||||
|
plugin verifies its expected per-project artifacts exist (adr: ADR system present; vault:
|
||||||
|
project hub note exists — optionally injecting it as context). All good → silent. Problem →
|
||||||
|
one short banner prompting corrective action ("<xyz> missing — run /os-vault:… to fix").
|
||||||
|
Plugins stay independent; new plugins get picked up automatically by following the
|
||||||
|
convention; only the glue plugin is mandatory for the collection.
|
||||||
|
|
||||||
|
This is architecture → **OpenSpec change + ADR** (use `openspec-propose`; decisions land in
|
||||||
|
`docs/memory-system/03-architecture-decisions.md`). Do not build outside that workflow.
|
||||||
|
|
||||||
|
## User's sketch (the starting design)
|
||||||
|
|
||||||
|
Per-plugin Python status scripts following a cc-os convention (naming, interface,
|
||||||
|
location), living in a specific gitignored hidden directory in the project. SessionStart
|
||||||
|
fires a cc-os master status script that enumerates all status scripts in that directory
|
||||||
|
(excluding itself) and runs them. Adding a plugin later = drop in a conforming script; no
|
||||||
|
global rewrites.
|
||||||
|
|
||||||
|
## The one design tension the proposal must settle (pressure-test, don't fiat)
|
||||||
|
|
||||||
|
**Where does status-check CODE live?**
|
||||||
|
|
||||||
|
- **Option A (user sketch): project-local dir** (e.g. `.cc-os/status.d/*.py`, gitignored).
|
||||||
|
Pro: per-project opt-in/customization. Con: code copies drift across projects; needs an
|
||||||
|
install/update step per project per plugin; the 2026-07-04 stale-cache incident shows
|
||||||
|
copy-drift is a real failure class here.
|
||||||
|
- **Option B: plugin-source enumeration** — each plugin ships `status/check.py` in its own
|
||||||
|
source; the master enumerates installed `os-*` plugins at session start. Same modularity
|
||||||
|
and zero-global-rewrite extension; no copies, no drift, no install step. Project-local
|
||||||
|
`.cc-os/` still exists but holds STATE only (suppress files, per-project config like
|
||||||
|
"this project opts out of hub-note checks"), mirroring os-adr's `.os-adr/` and
|
||||||
|
os-doc-hygiene's `.dochygiene/` — consolidating those is itself a candidate scope item.
|
||||||
|
|
||||||
|
Run `perspectives:simplifier` and `perspectives:implementer` (both sonnet) against both
|
||||||
|
shapes inside the proposal. Lean: B for code, A-style dir for state.
|
||||||
|
|
||||||
|
## Interface contract (draft for the proposal)
|
||||||
|
|
||||||
|
- Each check: stdin/argv gets project root + plugin config; stdout JSON
|
||||||
|
`{"status": "ok"|"warn", "message": str, "context": str|null}`; hard timeout (~5s);
|
||||||
|
non-zero exit or timeout = treated as warn with a generic message, never blocks session.
|
||||||
|
- **Silent on ok** — os-adr's near-zero-token SessionStart discipline is the template.
|
||||||
|
Master aggregates warns into ONE short banner; per-project once-per-day snooze +
|
||||||
|
permanent suppress via state dir (pattern already proven in os-adr/os-doc-hygiene).
|
||||||
|
- `context` field = optional session-context injection (the "pull the hub note into
|
||||||
|
context if useful" idea) — but gate it through injection-economics thinking (see the
|
||||||
|
design-template skill's filter); default null.
|
||||||
|
|
||||||
|
## Glue plugin scope ("cc-os collection manager")
|
||||||
|
|
||||||
|
Naming: per `~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` (read before
|
||||||
|
naming; `os-[domain]`, verb-first skills, NO `name:` frontmatter). Candidates: `os-core`
|
||||||
|
(lean) or `os-status` (narrower). Settle in the proposal.
|
||||||
|
|
||||||
|
Phase 1 (this change): master SessionStart hook + enumeration + interface contract +
|
||||||
|
convert the two existing consumers:
|
||||||
|
- os-adr: refactor its existing existence-check hook into a conforming status check (it is
|
||||||
|
already exactly this pattern — present→usage note, absent→once-per-day suggestion).
|
||||||
|
- os-vault: new hub-note check (project hub note exists for the current project; missing →
|
||||||
|
prompt to run onboarding/creation; optionally inject on present).
|
||||||
|
|
||||||
|
Phase 2 (separate change, later): collection-manager duties — verify installed cc-os
|
||||||
|
plugin caches are fresh vs source (would have auto-caught the 2026-07-04 stale-cache
|
||||||
|
incident), migrations complete, conventions followed (e.g. no `name:` in SKILL.md
|
||||||
|
frontmatter). Effectively an automated `bin/refresh-plugins` advisor.
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
1. OpenSpec change (design.md resolving the tension above + specs + tasks) — via
|
||||||
|
`openspec-propose`.
|
||||||
|
2. ADR in `docs/memory-system/03-architecture-decisions.md` (status-check convention +
|
||||||
|
code-location decision).
|
||||||
|
3. Implementation via `openspec-apply-change`: master hook + shared contract module
|
||||||
|
(Python, matching os-vault's deep-module style: thin entry points, shared config/io
|
||||||
|
modules), conforming checks for os-adr + os-vault, model-free tests (Python, mirroring
|
||||||
|
`tests/hook_test.py` style) + invariants.md.
|
||||||
|
4. `bin/refresh-plugins`, `claude plugin details` verification, fresh-session smoke test.
|
||||||
|
|
||||||
|
## Models
|
||||||
|
|
||||||
|
Design synthesis + ADR: orchestrator. Perspective reviews: sonnet. Implementation: haiku
|
||||||
|
for thin scripts/tests, sonnet for the shared contract module. No behavioral eval needed —
|
||||||
|
deterministic code; invariants + model-free tests suffice (same rationale as the os-adr
|
||||||
|
hook).
|
||||||
Loading…
Reference in New Issue