# Plan set: second-batch plugin evals + status-check convention

_Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session._

Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's
2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared
discipline only.

## Workstreams

1. **WS1 — os-orchestration session audit** → `ws1-orchestration-audit.md`
   Evidence-gathering over real session transcripts; produces failure-mode clusters and an
   eval-scenario backlog. No design dependencies — start anytime.
   **DONE 2026-07-06:** findings in `docs/orchestration-audit/2026-07-06-findings.md`
   (4 verified clusters, E1–E4 scenario backlog); extractor at
   `plugins/os-orchestration/audit/bin/extract`.
2. **WS2 — os-vault write-behavior eval** → `ws2-os-vault-write-eval.md`
   Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately.
   The eval harness itself should start after WS3's design lands (hub-note automation is a
   status-check consumer) and after the quick wins (baseline measures post-bugfix behavior).
3. **WS3 — status-check convention + glue plugin** → `ws3-status-convention-plugin.md`
   Architecture: OpenSpec change + ADR. Runs in parallel with WS1.

## Recommended sequencing

- Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis).
- Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins.
- Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording
  loop is needed.

## Shared discipline (from the first batch — non-negotiable)

Before designing or running ANY eval below, Read:

- `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` (run modes, cache
  refresh, canary cell, reps, pipefail/rescore gotchas)
- `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, run-set
  vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot)
- `~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md` (production audit → new
  scenarios)

Standing rules: held-out Task blocks are never run informally; all reps counted (no
discards); orchestrating session acts as final judge — verify every agent claim against
primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks.