cc-os/docs/implementation-status/os-context.md

127 lines
9.6 KiB
Markdown
Raw Permalink Normal View History

# os-context (formerly os-orchestration) — component detail
_Leaf file of [../implementation-status.md](../implementation-status.md). Read on demand._
**Global os-context plugin** — `cc-os/plugins/os-context/` (git-tracked); symlinked into
`~/.claude/plugins/os-context`. Migrated 2026-07-03 from a standalone repo
(`~/dev/cc-plugins/orchestration/`), superseding per-project orchestration text (ADR-0019);
renamed and generalized from os-orchestration 2026-07-13 (ADR-0031). Entries below predating
the rename refer to the plugin by its former name.
## Component inventory
- Hooks: `hooks/inject.py` — SessionStart prompt composer: globs
`prompts/session-start/*.md`, sorts by filename, concatenates into one additionalContext
block for all sessions, with a deterministic line-count budget (warn >120 lines, refuse
>240 lines). Byte-identical output to the old single-file `ORCHESTRATION.md` behavior when
only `10-orchestration.md` is present (tested).
- Prompt files (2026-07-13): `10-orchestration.md` (the canonical session-orchestration +
delegation-economics rules; formerly `ORCHESTRATION.md`), `20-kickoff-conventions.md`,
`30-standing-safety.md`, `40-decision-memos.md` (43 lines; total injection 102/120).
Decision-memo auto-memory file deleted per single-source rule.
- Skill: `/os-context:audit-sessions` (formerly `/os-orchestration:audit-sessions`).
- State: `~/.local/state/os-context/` (old `os-orchestration` path symlinked;
`OS_ORCH_AUDIT_LEDGER` env var name kept). os-status's `orchestration-audit-due` check
updated to match.
## Session audits and wording iterations
- **WS1 session audit (2026-07-06):** 10 stratified real sessions audited via
`audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) +
parallel sonnet auditors + verified synthesis. Findings + E1E4 eval-scenario backlog:
`docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param silently
overridden to haiku on all 23 verified spawns — root cause
`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (removed
2026-07-06); omitted model param inherits the main-loop model; no over-sharing/full-dump
failures anywhere.
- **Wording fixes shipped (2026-07-06, post-audit):** ORCHESTRATION.md rewritten with
trigger-conditioned when→then phrasing for the three verified clusters — explicit
`model:` before every Agent call, subagent model self-report on sonnet/opus spawns (the E1
canary proved `resolvedModel` is NOT model-visible in the launch result; self-report is
the only observable channel), don't re-cover your own ground before delegating.
Delegation thresholds unchanged. Deterministic env-override detection delegated to
os-status (WS3).
- **Eval harness E1E3 (2026-07-06):** `plugins/os-context/eval/` — scripted headless
orchestration-behavior eval scored by the audit extractor (extended with per-segment read
targets). 6 run-set scenarios (E1P/N downgrade detection with runner stub; E2P/N
explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs
orienting-reads-correct) + 6 frozen reserve twins; Node.js "relaystation" fixture with
deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`,
narrow frozen haiku-judge fallback for E1 language axes only, stubbable via
`ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21. **Scenario Task blocks are
held-out — never run informally.** Procedure: `plugins/os-context/eval/README.md`.
- **Baseline grid (2026-07-06,** run-set × sonnet/haiku × 3 reps + 2 counted canaries):
negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims);
positives 1/18 on delegate-at-all — but transcript verification shows most positive FAILs
are *superior direct strategies* (scripted bulk edit; surgical greps over 400KB logs),
which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes
(haiku: whole 112KB logs into own context; 20-call per-file grind). 14/14 spawns
explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report.
E2P/E3P A-axis criterion redesign pre-registered; reserve stayed frozen. ORCHESTRATION.md
gained a scripted-bulk-edit carve-out. TSV: `eval/results/2026-07-06-baseline-grid.tsv`;
analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
- **WS4 economics loop COMPLETE + SHIPPED (2026-07-08;** run-set AND reserve now
contaminated for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry +
tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger
lists, batching rule (~58 items per grouped spawn, SendMessage reuse), async rule,
Workflow effort dial; explicit-model/self-report/don't-re-cover-ground rules unchanged.
Harness extended first (commit b226601): real token-econ axes — subagent transcripts
persist at `<projects>/<flat-cwd>/<session-id>/subagents/agent-*.jsonl` with per-message
usage blocks (vault note [[claude-code-subagent-transcripts-and-token-accounting]]) — E5
batching pair + reserve twins (subagent-authored, held out), fable column, pre-registered
E2P/E3P criterion redesign landed. Results: E5P 0/6 → majority-PASS both tiers incl.
reserve twin 5/5 (fable median mltok 43% of baseline); negatives clean everywhere
(36/36). Residual gaps: E1P sonnet delegate-at-all (fixture too small), fable
downgrade-flag ~1/4 judge flicker. 15 truncated baseline reps mechanically
excluded+replaced (limit-banner sweep promoted to the autoresearch howto). TSVs:
`eval/results/2026-07-08-*.tsv`; loop log: `autoresearch/classic-260708-1039/`; vault
note [[os-orchestration-ws4-econ-wording-results]].
- **First IRL re-audit DONE (2026-07-10, ahead of schedule):** 19 production transcripts
post-2026-07-08 audited by 3 parallel agents (run: `autoresearch/improve-260710-1057/`).
Solved surfaces: model-param discipline (~48/49 spawns pinned), over-delegation guardrail.
Tier 1 findings → same-day one-shot ORCHESTRATION.md production patch (grid validation
impossible — sets contaminated): settled-design mid-session fan-out trigger,
never-sleep-poll rule, own eval/benchmark loops named as mechanical sequences,
unfamiliar-tree ~5-call Explore trigger. Acceptance check = next IRL audit (~23 weeks;
baseline: settled-design drift 8/19 sessions). Side finding needing separate repro:
harness ignored explicit `model: "sonnet"` pins on general-purpose agents (env override
unset — distinct from WS1 cause); the self-report line caught it.
- **IRL audit semi-automated (2026-07-10):** recurring biweekly audit formalized, working
from any cwd. Deterministic layer: `audit/bin/audit-stats` (Ruby, composes
`audit/bin/extract` fact-sheets) enumerates production transcripts since the last run
(excludes sandboxes, `*-eval` dirs, active/trivial sessions), writes
`~/.local/state/os-context/audit/run-<date>/` (stats.tsv/flags.md/sheets) and appends the
trend ledger `metrics.tsv` (seeded 2026-07-10: 23 sessions, 52 spawns, 1 missing model).
Judgment layer: `/os-context:audit-sessions` (rubric in `references/`) — sonnet auditor
fan-out over flagged regions only, tiered synthesis, run-over-run trend verdicts, stops
before any wording edit. Nudge: os-status check `orchestration-audit-due` warns when the
ledger's last row is ≥14 days old (missing ledger = machine not opted in, silent; env
override `OS_ORCH_AUDIT_LEDGER` for tests). Planka recurrence-manifest entry skipped
(gem offsets are `+1y/+6m/+3m/+1m` only; a `+2w` offset is blocked on parked gem
publish/versioning decisions).
- **Audit-driven wording patch (2026-07-13):** IRL session audit (session fdee577b) found
the delegation-economics rules failed to fire when a main-loop fable model implemented a
fully-specified fix list itself via TDD (~110k tokens). Two approved additions to
`prompts/session-start/10-orchestration.md`: (1) names a red-green TDD loop over an
already-specified fix list as a single delegable unit (issue #29); (2) names
review/diagnosis-agent convergence on a confirmed defect list as a fresh delegation
decision, never a continuation of the main loop's review thread (issue #30). Applied
directly (design pre-approved, no wording-loop re-run); `bin/refresh-plugins` run after
the edit.
- **Absolute delegation policy (2026-07-17, ADR-0043):** IRL transcript review showed the
per-task delegation threshold fails by induction — chain length is unknowable at decision
time, each "short chain" call is locally defensible, and sessions accrete 100k+ tokens of
main-loop tool results. `10-orchestration.md` rewritten (design pre-approved in session):
executive frame, enumerated E-a..E-d exemptions (no evaluative carve-outs), one persistent
sonnet manager continued via SendMessage, opus escalation on observed failure only.
Audit rubric rewritten to match, shaped by a sonnet perspectives fan-out (measurement +
devils-advocate): exemption decision procedure with recorded steps per finding, new
`exemption-chaining` category, `over-delegation` narrowed to `delegation-overhead-mismatch`
(not deleted), old defenses demoted to LOW/`accepted-cost` severity modifiers,
`missed-delegation/read-write` sub-tag. `audit/bin/extract` gains `read_write_profile`
(total main-loop Read/Write/Edit incl. non-consecutive — largest prior session: 67 total,
only 14 visible to the run detector) and drops its stale pro-direct-work trailer. Eval
E1E3/E5 retired as a measurement instrument (README banner): sets contaminated AND
scenarios encode the superseded policy; kept as harness template. Validation of the new
wording comes only from future IRL audits.