jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/docs/implementation-status/os-context.md

9.6 KiB
Raw Blame History

os-context (formerly os-orchestration) — component detail

Leaf file of ../implementation-status.md. Read on demand.

Global os-context plugin — cc-os/plugins/os-context/ (git-tracked); symlinked into ~/.claude/plugins/os-context. Migrated 2026-07-03 from a standalone repo (~/dev/cc-plugins/orchestration/), superseding per-project orchestration text (ADR-0019); renamed and generalized from os-orchestration 2026-07-13 (ADR-0031). Entries below predating the rename refer to the plugin by its former name.

Component inventory

  • Hooks: hooks/inject.py — SessionStart prompt composer: globs prompts/session-start/*.md, sorts by filename, concatenates into one additionalContext block for all sessions, with a deterministic line-count budget (warn >120 lines, refuse

    240 lines). Byte-identical output to the old single-file ORCHESTRATION.md behavior when only 10-orchestration.md is present (tested).

  • Prompt files (2026-07-13): 10-orchestration.md (the canonical session-orchestration + delegation-economics rules; formerly ORCHESTRATION.md), 20-kickoff-conventions.md, 30-standing-safety.md, 40-decision-memos.md (43 lines; total injection 102/120). Decision-memo auto-memory file deleted per single-source rule.
  • Skill: /os-context:audit-sessions (formerly /os-orchestration:audit-sessions).
  • State: ~/.local/state/os-context/ (old os-orchestration path symlinked; OS_ORCH_AUDIT_LEDGER env var name kept). os-status's orchestration-audit-due check updated to match.

Session audits and wording iterations

  • WS1 session audit (2026-07-06): 10 stratified real sessions audited via audit/bin/extract (Ruby fact-sheet extractor, dual-use as future eval checker) + parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog: docs/orchestration-audit/2026-07-06-findings.md. Headlines: model param silently overridden to haiku on all 23 verified spawns — root cause CLAUDE_CODE_SUBAGENT_MODEL=haiku in ~/.claude/settings.json env block (removed 2026-07-06); omitted model param inherits the main-loop model; no over-sharing/full-dump failures anywhere.
  • Wording fixes shipped (2026-07-06, post-audit): ORCHESTRATION.md rewritten with trigger-conditioned when→then phrasing for the three verified clusters — explicit model: before every Agent call, subagent model self-report on sonnet/opus spawns (the E1 canary proved resolvedModel is NOT model-visible in the launch result; self-report is the only observable channel), don't re-cover your own ground before delegating. Delegation thresholds unchanged. Deterministic env-override detection delegated to os-status (WS3).
  • Eval harness E1–E3 (2026-07-06): plugins/os-context/eval/ — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (eval/bin/check, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via ORCH_EVAL_JUDGE_CMD); model-free bin/self-test 21/21. Scenario Task blocks are held-out — never run informally. Procedure: plugins/os-context/eval/README.md.
  • Baseline grid (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on delegate-at-all — but transcript verification shows most positive FAILs are superior direct strategies (scripted bulk edit; surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (haiku: whole 112KB logs into own context; 20-call per-file grind). 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. E2P/E3P A-axis criterion redesign pre-registered; reserve stayed frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out. TSV: eval/results/2026-07-06-baseline-grid.tsv; analysis: vault note os-orchestration-eval-baseline-grid-results.
  • WS4 economics loop COMPLETE + SHIPPED (2026-07-08; run-set AND reserve now contaminated for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry + tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger lists, batching rule (~5–8 items per grouped spawn, SendMessage reuse), async rule, Workflow effort dial; explicit-model/self-report/don't-re-cover-ground rules unchanged. Harness extended first (commit b226601): real token-econ axes — subagent transcripts persist at <projects>/<flat-cwd>/<session-id>/subagents/agent-*.jsonl with per-message usage blocks (vault note claude-code-subagent-transcripts-and-token-accounting) — E5 batching pair + reserve twins (subagent-authored, held out), fable column, pre-registered E2P/E3P criterion redesign landed. Results: E5P 0/6 → majority-PASS both tiers incl. reserve twin 5/5 (fable median mltok 43% of baseline); negatives clean everywhere (36/36). Residual gaps: E1P sonnet delegate-at-all (fixture too small), fable downgrade-flag ~1/4 judge flicker. 15 truncated baseline reps mechanically excluded+replaced (limit-banner sweep promoted to the autoresearch howto). TSVs: eval/results/2026-07-08-*.tsv; loop log: autoresearch/classic-260708-1039/; vault note os-orchestration-ws4-econ-wording-results.
  • First IRL re-audit DONE (2026-07-10, ahead of schedule): 19 production transcripts post-2026-07-08 audited by 3 parallel agents (run: autoresearch/improve-260710-1057/). Solved surfaces: model-param discipline (~48/49 spawns pinned), over-delegation guardrail. Tier 1 findings → same-day one-shot ORCHESTRATION.md production patch (grid validation impossible — sets contaminated): settled-design mid-session fan-out trigger, never-sleep-poll rule, own eval/benchmark loops named as mechanical sequences, unfamiliar-tree ~5-call Explore trigger. Acceptance check = next IRL audit (~2–3 weeks; baseline: settled-design drift 8/19 sessions). Side finding needing separate repro: harness ignored explicit model: "sonnet" pins on general-purpose agents (env override unset — distinct from WS1 cause); the self-report line caught it.
  • IRL audit semi-automated (2026-07-10): recurring biweekly audit formalized, working from any cwd. Deterministic layer: audit/bin/audit-stats (Ruby, composes audit/bin/extract fact-sheets) enumerates production transcripts since the last run (excludes sandboxes, *-eval dirs, active/trivial sessions), writes ~/.local/state/os-context/audit/run-<date>/ (stats.tsv/flags.md/sheets) and appends the trend ledger metrics.tsv (seeded 2026-07-10: 23 sessions, 52 spawns, 1 missing model). Judgment layer: /os-context:audit-sessions (rubric in references/) — sonnet auditor fan-out over flagged regions only, tiered synthesis, run-over-run trend verdicts, stops before any wording edit. Nudge: os-status check orchestration-audit-due warns when the ledger's last row is ≥14 days old (missing ledger = machine not opted in, silent; env override OS_ORCH_AUDIT_LEDGER for tests). Planka recurrence-manifest entry skipped (gem offsets are +1y/+6m/+3m/+1m only; a +2w offset is blocked on parked gem publish/versioning decisions).
  • Audit-driven wording patch (2026-07-13): IRL session audit (session fdee577b) found the delegation-economics rules failed to fire when a main-loop fable model implemented a fully-specified fix list itself via TDD (~110k tokens). Two approved additions to prompts/session-start/10-orchestration.md: (1) names a red-green TDD loop over an already-specified fix list as a single delegable unit (issue #29); (2) names review/diagnosis-agent convergence on a confirmed defect list as a fresh delegation decision, never a continuation of the main loop's review thread (issue #30). Applied directly (design pre-approved, no wording-loop re-run); bin/refresh-plugins run after the edit.
  • Absolute delegation policy (2026-07-17, ADR-0043): IRL transcript review showed the per-task delegation threshold fails by induction — chain length is unknowable at decision time, each "short chain" call is locally defensible, and sessions accrete 100k+ tokens of main-loop tool results. 10-orchestration.md rewritten (design pre-approved in session): executive frame, enumerated E-a..E-d exemptions (no evaluative carve-outs), one persistent sonnet manager continued via SendMessage, opus escalation on observed failure only. Audit rubric rewritten to match, shaped by a sonnet perspectives fan-out (measurement + devils-advocate): exemption decision procedure with recorded steps per finding, new exemption-chaining category, over-delegation narrowed to delegation-overhead-mismatch (not deleted), old defenses demoted to LOW/accepted-cost severity modifiers, missed-delegation/read-write sub-tag. audit/bin/extract gains read_write_profile (total main-loop Read/Write/Edit incl. non-consecutive — largest prior session: 67 total, only 14 visible to the run detector) and drops its stale pro-direct-work trailer. Eval E1–E3/E5 retired as a measurement instrument (README banner): sets contaminated AND scenarios encode the superseded policy; kept as harness template. Validation of the new wording comes only from future IRL audits.