jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/docs/plans/2026-07-06-plugin-evals-ove...

2.4 KiB
Raw Blame History

Plan set: second-batch plugin evals + status-check convention

Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session.

Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's 2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared discipline only.

Workstreams

  1. WS1 — os-orchestration session audit → ws1-orchestration-audit.md Evidence-gathering over real session transcripts; produces failure-mode clusters and an eval-scenario backlog. No design dependencies — start anytime. DONE 2026-07-06: findings in docs/orchestration-audit/2026-07-06-findings.md (4 verified clusters, E1–E4 scenario backlog); extractor at plugins/os-orchestration/audit/bin/extract.
  2. WS2 — os-vault write-behavior eval → ws2-os-vault-write-eval.md Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately. The eval harness itself should start after WS3's design lands (hub-note automation is a status-check consumer) and after the quick wins (baseline measures post-bugfix behavior).
  3. WS3 — status-check convention + glue plugin → ws3-status-convention-plugin.md Architecture: OpenSpec change + ADR. Runs in parallel with WS1.

Recommended sequencing

  • Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis).
  • Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins.
  • Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording loop is needed.

Shared discipline (from the first batch — non-negotiable)

Before designing or running ANY eval below, Read:

  • ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md (run modes, cache refresh, canary cell, reps, pipefail/rescore gotchas)
  • ~/Documents/SecondBrain/eval-methodology-ladder.md (paired positives/negatives, run-set vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot)
  • ~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md (production audit → new scenarios)

Standing rules: held-out Task blocks are never run informally; all reps counted (no discards); orchestrating session acts as final judge — verify every agent claim against primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks.