2026-07-06 17:54:14 +00:00
|
|
|
|
# Plan set: second-batch plugin evals + status-check convention
|
|
|
|
|
|
|
|
|
|
|
|
_Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session._
|
|
|
|
|
|
|
|
|
|
|
|
Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's
|
|
|
|
|
|
2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared
|
|
|
|
|
|
discipline only.
|
|
|
|
|
|
|
|
|
|
|
|
## Workstreams
|
|
|
|
|
|
|
|
|
|
|
|
1. **WS1 — os-orchestration session audit** → `ws1-orchestration-audit.md`
|
|
|
|
|
|
Evidence-gathering over real session transcripts; produces failure-mode clusters and an
|
|
|
|
|
|
eval-scenario backlog. No design dependencies — start anytime.
|
2026-07-06 21:34:42 +00:00
|
|
|
|
**DONE 2026-07-06:** findings in `docs/orchestration-audit/2026-07-06-findings.md`
|
|
|
|
|
|
(4 verified clusters, E1–E4 scenario backlog); extractor at
|
|
|
|
|
|
`plugins/os-orchestration/audit/bin/extract`.
|
2026-07-06 17:54:14 +00:00
|
|
|
|
2. **WS2 — os-vault write-behavior eval** → `ws2-os-vault-write-eval.md`
|
|
|
|
|
|
Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately.
|
|
|
|
|
|
The eval harness itself should start after WS3's design lands (hub-note automation is a
|
|
|
|
|
|
status-check consumer) and after the quick wins (baseline measures post-bugfix behavior).
|
|
|
|
|
|
3. **WS3 — status-check convention + glue plugin** → `ws3-status-convention-plugin.md`
|
|
|
|
|
|
Architecture: OpenSpec change + ADR. Runs in parallel with WS1.
|
|
|
|
|
|
|
|
|
|
|
|
## Recommended sequencing
|
|
|
|
|
|
|
|
|
|
|
|
- Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis).
|
|
|
|
|
|
- Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins.
|
|
|
|
|
|
- Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording
|
|
|
|
|
|
loop is needed.
|
|
|
|
|
|
|
|
|
|
|
|
## Shared discipline (from the first batch — non-negotiable)
|
|
|
|
|
|
|
|
|
|
|
|
Before designing or running ANY eval below, Read:
|
|
|
|
|
|
|
|
|
|
|
|
- `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` (run modes, cache
|
|
|
|
|
|
refresh, canary cell, reps, pipefail/rescore gotchas)
|
|
|
|
|
|
- `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, run-set
|
|
|
|
|
|
vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot)
|
|
|
|
|
|
- `~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md` (production audit → new
|
|
|
|
|
|
scenarios)
|
|
|
|
|
|
|
|
|
|
|
|
Standing rules: held-out Task blocks are never run informally; all reps counted (no
|
|
|
|
|
|
discards); orchestrating session acts as final judge — verify every agent claim against
|
|
|
|
|
|
primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks.
|
