45 lines
2.4 KiB
Markdown
45 lines
2.4 KiB
Markdown
# Plan set: second-batch plugin evals + status-check convention
|
||
|
||
_Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session._
|
||
|
||
Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's
|
||
2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared
|
||
discipline only.
|
||
|
||
## Workstreams
|
||
|
||
1. **WS1 — os-orchestration session audit** → `ws1-orchestration-audit.md`
|
||
Evidence-gathering over real session transcripts; produces failure-mode clusters and an
|
||
eval-scenario backlog. No design dependencies — start anytime.
|
||
**DONE 2026-07-06:** findings in `docs/orchestration-audit/2026-07-06-findings.md`
|
||
(4 verified clusters, E1–E4 scenario backlog); extractor at
|
||
`plugins/os-orchestration/audit/bin/extract`.
|
||
2. **WS2 — os-vault write-behavior eval** → `ws2-os-vault-write-eval.md`
|
||
Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately.
|
||
The eval harness itself should start after WS3's design lands (hub-note automation is a
|
||
status-check consumer) and after the quick wins (baseline measures post-bugfix behavior).
|
||
3. **WS3 — status-check convention + glue plugin** → `ws3-status-convention-plugin.md`
|
||
Architecture: OpenSpec change + ADR. Runs in parallel with WS1.
|
||
|
||
## Recommended sequencing
|
||
|
||
- Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis).
|
||
- Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins.
|
||
- Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording
|
||
loop is needed.
|
||
|
||
## Shared discipline (from the first batch — non-negotiable)
|
||
|
||
Before designing or running ANY eval below, Read:
|
||
|
||
- `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` (run modes, cache
|
||
refresh, canary cell, reps, pipefail/rescore gotchas)
|
||
- `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, run-set
|
||
vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot)
|
||
- `~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md` (production audit → new
|
||
scenarios)
|
||
|
||
Standing rules: held-out Task blocks are never run informally; all reps counted (no
|
||
discards); orchestrating session acts as final judge — verify every agent claim against
|
||
primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks.
|