From 02b330df2c37cce596e6c6c3e8fe0268d25436b6 Mon Sep 17 00:00:00 2001 From: jared Date: Wed, 8 Jul 2026 11:22:45 -0400 Subject: [PATCH] vault: session notes 2026-07-08 --- claude-md-budget-linter-plugin-concept.md | 79 ++++++++++++ eval-methodology-ladder.md | 3 +- howto/running-autoresearch-skill-evals.md | 3 +- os-orchestration-ws4-econ-wording-results.md | 129 +++++++++++++++++++ 4 files changed, 212 insertions(+), 2 deletions(-) create mode 100644 claude-md-budget-linter-plugin-concept.md create mode 100644 os-orchestration-ws4-econ-wording-results.md diff --git a/claude-md-budget-linter-plugin-concept.md b/claude-md-budget-linter-plugin-concept.md new file mode 100644 index 0000000..658e431 --- /dev/null +++ b/claude-md-budget-linter-plugin-concept.md @@ -0,0 +1,79 @@ +--- +type: plan +title: "CLAUDE.md budget linter plugin — concept (deterministic hook + exception config + maintenance agent)" +summary: "Future cc-os plugin concept for keeping CLAUDE.md (and eventually whole AI projects) within deterministic token/size budgets — a hook-driven linter that stays silent in range, escalates to a maintenance agent when out of range, and records user-agreed exceptions in a per-project config so allowed growth stays bounded." +tags: + - type/plan + - tool/claude-code + - domain/context-engineering + - project/cc-os +scope: global +last_updated: 2026-07-08 +date: 2026-07-08 +related: + - "[[claude-code-system-prompt-customization-surface]]" +--- + +# CLAUDE.md budget linter plugin — concept + +Status: idea only, deliberately out of scope as of 2026-07-08. Revisit after the +system-prompt-profiles work (`cc-os/docs/plans/system-prompt-profiles-assessment.md`). +Source discussion: cc-os session 2026-07-08; maintenance philosophy drawn from +`~/servers/proxmox-hermes/docs/claude-md-maintenance.md`. + +## Problem + +- CLAUDE.md degrades silently: bloat, stale facts, and instruction overload collapse rule + adherence (community guidance: ~100 lines of instruction; adherence degrades uniformly + past ~150–200 instructions). Measured example: cc-os CLAUDE.md reached ~11k tokens — + ~10x a sane budget — by accreting an implementation-status changelog. +- Pruning is currently manual and memory-dependent ("review it monthly") — exactly the kind + of discipline that erodes. Nothing enforces a ceiling, and one "just this once" exception + becomes unbounded growth because the exception is never quantified. +- The same failure mode applies project-wide: docs that outgrow progressive disclosure, + copies instead of pointers, files nobody re-reads. + +## Goal + +Deterministic, config-driven enforcement of context budgets — zero AI tokens on the happy +path — with a human-in-the-loop escalation only when a budget is exceeded. Exceptions are +explicit, quantified, and re-checked, so agreed growth stays bounded instead of becoming +the new normal. + +## How it works (mid-level) + +1. **Hook → linter.** A SessionStart (or periodic) hook runs a deterministic linter over + CLAUDE.md. In range → no output, no action, 100% deterministic, no model call. +2. **Per-project dot-config** (JSON for easy slurping, e.g. `.claude-md-lint.json` or a key + in an existing `.cc-os/config`): default budget (e.g. 1k tokens), cadence + (`check_every_days`), and an **exceptions block** — each exception records the agreed + limit, the specific reason, and who/when agreed (e.g. "allowed 1.2k because X, agreed + 2026-07-08"). The linter checks against the *effective* limit, so an exception raises + the ceiling without removing it. +3. **Out of range → maintenance agent.** Only then does an AI step run: it assesses the + file (stale sections, changelog accretion, copies-vs-pointers, `@`-import opportunities) + and proposes suggestions to the user. +4. **User resolves**: accept the pruning suggestions, push back, or codify a new quantified + exception in the config. Never auto-edit CLAUDE.md — automated writes are the known + failure mode (noise accumulation, contradictory rules, safety-rule drift). +5. **Cadence control**: config decides whether the hook checks every session or every N + days (stamp in a gitignored state dir, same pattern as os-doc-hygiene's `.dochygiene/`). + +## Possible extension: whole-project checks + +The same skeleton (deterministic scan → config budgets → agent escalation) generalizes to +weekly project hygiene: file-size ceilings, progressive-disclosure usage (pointers vs +inlined copies), stale-doc detection, dead `@`-imports. + +## Build-time considerations + +- **Overlap with os-doc-hygiene is significant** — it already does deterministic + SessionStart reminders, per-project gitignored state, AI-assisted classification, and + git-safe cleanup for stale/bloated docs. First design question at build time: is this a + new `os-*` plugin or a CLAUDE.md-specific check + budget-config feature *inside* + os-doc-hygiene (and/or an os-status check for "CLAUDE.md over budget")? +- Read `cc-os-plugin-skill-naming-convention.md` before naming anything. +- Token measurement should be deterministic and cheap (chars/4 heuristic or a local + tokenizer — pick one and freeze it; the budget is a contract, so the measure can't drift). +- Eval approach exists: same deterministic-first checker + ladder playbook as the other + os-* plugins. diff --git a/eval-methodology-ladder.md b/eval-methodology-ladder.md index 9dfc7ab..d25e43e 100644 --- a/eval-methodology-ladder.md +++ b/eval-methodology-ladder.md @@ -9,7 +9,7 @@ tags: - convention/eval-design - tool/autoresearch scope: global -last_updated: 2026-07-07 # added lesson-promotion pattern (eval-results → methodology notes, Loop Inputs handoff) +last_updated: 2026-07-08 # econ-guard anti-pattern (WS4) date: 2026-07-06 related: - running-autoresearch-skill-evals @@ -95,6 +95,7 @@ artifact that lets a later reader diff proposal against what shipped. - **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold. - **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set. - **A checker that contradicts the shipped instructions.** Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-*disobedience*. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run. +- **Econ guards phrased as ratios of a grind baseline.** When a positive scenario's baseline behavior is an *efficient but wrong-shaped* strategy (e.g. a cheap main-loop grind), a "candidate must stay ≤N% of baseline cost" guard punishes the correct behavior the checker itself demands — delegation legitimately costs more than grinding on a cheap tier. Set cheap-tier econ guards in absolute tokens, or condition them on whether the task is scriptable; reduction targets belong only on the expensive-tier cells whose baseline IS the miss (os-orchestration WS4, 2026-07-08: sonnet guard breached at 168% for identical task completion, accepted with rationale). - **Trusting model-free self-tests to validate checker *semantics*.** Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of *shipped-instruction-compliant* behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal. ## Known Limitations diff --git a/howto/running-autoresearch-skill-evals.md b/howto/running-autoresearch-skill-evals.md index 15debde..3276570 100644 --- a/howto/running-autoresearch-skill-evals.md +++ b/howto/running-autoresearch-skill-evals.md @@ -9,7 +9,7 @@ tags: - tool/autoresearch - project/cc-os scope: global -last_updated: 2026-07-06 +last_updated: 2026-07-08 date: 2026-07-04 related: - os-adr-eval-b-grid-results-and-observations @@ -98,6 +98,7 @@ salience in the hook note / find-skill description). One "failure" label, two di - **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep. - **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement. - **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result. +- **Account session limits truncate headless reps into plausible verdicts** — a `claude -p` rep cut by a usage/session limit still yields a scoreable transcript: positives FAIL with real-looking axes (task half done) and *negatives can wrongly PASS* (session died before the forbidden action). Sweep EVERY rep's final assistant message for the limit banner ("You've hit your session limit") before counting; excluded reps are infra errors, mechanically identified independent of verdict, so replacing them with fresh reps is not peeking (WS4, 2026-07-08: 15/40 grid reps excluded this way). - **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it. - **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06). - **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias. diff --git a/os-orchestration-ws4-econ-wording-results.md b/os-orchestration-ws4-econ-wording-results.md new file mode 100644 index 0000000..ba7909f --- /dev/null +++ b/os-orchestration-ws4-econ-wording-results.md @@ -0,0 +1,129 @@ +--- +type: eval-results +title: "os-orchestration WS4 economics wording loop (ORCHESTRATION.md v3, fable+sonnet)" +summary: "The v3 cost-asymmetry/symmetric-trigger/batching wording flips E5P implementation fan-out from 0/6 to majority-PASS on both tiers and cuts fable main-loop output tokens to 43% of baseline, with all negatives clean on run-set AND held-out reserve — shipped after one iteration + confirm + reserve; residual gaps are E1P sonnet non-delegation (fixture-scale) and a ~1/4 fable downgrade-flag judge flicker." +tags: + - type/eval-results + - domain/llm-evaluation + - tool/os-orchestration + - project/cc-os +scope: global +last_updated: 2026-07-08 +date: 2026-07-08 +related: + - os-orchestration-eval-baseline-grid-results + - orchestration-prompting-claude-5-era + - eval-methodology-ladder + - running-autoresearch-skill-evals + - claude-code-subagent-transcripts-and-token-accounting +source: cc-os +--- + +# os-orchestration WS4 economics wording loop + +## Results Grid/Threshold + +**Passing criterion:** per-cell ≥2/3 (sonnet) / 2/2 (fable) reps PASS on the frozen +deterministic checker; H3 econ thresholds pre-registered in +`docs/plans/ws4-orchestration-economics.md` (E5P fable median mltok ≤37,783 = 60% of +baseline; E3P non-regression guard; negatives must stay clean). + +Pre-wording baseline (2026-07-08, 44 counted reps) → candidate-1 (iter-1 15 reps + confirm +40 reps) → frozen reserve (40 reps). Zero harness errors; every rep swept for +session-limit truncation (15 baseline reps excluded+replaced by that mechanical criterion). + +| Cell | baseline | confirm (candidate) | reserve twin | +|---|---|---|---| +| E5P fable | 0/3 (grind, median mltok 62,971) | 2/2 (median across 5 candidate reps 27,372 = 43%) | 2/2 | +| E5P sonnet | 0/3 | 2/3 (+3/3 iter-1) | 3/3 | +| E1P fable | 1/2 | 1/2 (B judge flicker; 3/4 across iter+confirm) | 1/2 (A miss) | +| E1P sonnet | 0/3 | 0/3 | 0/3 | +| E2P / E3P (redesigned criteria) | 6/6 / 6/6 | 5/5 / 5/5 | 5/5 / 5/5 | +| all negatives (E1N/E2N/E3N/E5N) | clean | 16/16 | 20/20 | + +**Summary:** H1 (symmetric cost-framed triggers) and H2 (batching: ≤3 grouped spawns, one +round, explicit models — reserve fable rep used 2 explicit-haiku spawns) confirmed; H3 met +on the pre-registered E5P-fable axis (43% of baseline mltok). E1P sonnet never delegates at +any point (fixture too small to force it — pre-registered limitation); zero over-delegation +anywhere, including E5N sequential tasks under the new batching rule. + +## Measurement Setup + +**Hypotheses / Scenarios Tested** — H1/H2/H3 pre-registered in the WS4 plan; E5 +(batching/implementation fan-out, mirrors the 2026-07-08 mini-audit Write×13 exemplar) added +with reserve twins authored by a subagent so held-out content never entered the tuning +session's context. E2P/E3P criteria redesigned per the 2026-07-06 pre-registration (scripted- +direct PASS branch; 74KB S7 ingestion anchor + mechanical root-cause concepts). + +**Fixture and Sampling** — relaystation Node.js fixture + new handler-spec (12 pairwise- +distinct handler modules, deliberately non-templatable). sonnet ×3 / fable ×2 reps per cell; +fable at reduced reps for cost per plan. + +**Experimental Control (Frozen Surfaces)** — checker/extractor/runner/fixtures/scenarios/ +rubric frozen during the loop; ORCHESTRATION.md wording the only tuning surface; +`bin/refresh-plugins` + cache==source diff before every grid. One instrument fix landed +between confirm and reserve (E3P concepts keyed by scenario id — the reserve twin targets +gen-logs' other planted incident; fixed without reading reserve files, run-set scoring +byte-identical, self-test 34/34). + +## Validity and Limitations + +**How to Interpret These Results** — run-set AND reserve are now both contaminated for +future wording tuning (reserve was the final measurement). The reserve pass is the +generalization claim: same axes, different surface domain, never read informally. Econ +figures are real token sums (subagent transcripts persist under +`/subagents/agent-*.jsonl` — see [[claude-code-subagent-transcripts-and-token-accounting]]), +not the char-count proxy the mini-audit used. + +**Weaknesses of This Eval** — headless `-p` population may differ from interactive sessions +(unresolved since the 2026-07-06 baseline). E1P fixture too small to force delegation on +sonnet; its cell says nothing about capability. H3(c) sonnet econ guard (≤150% of grind +baseline) was breached at median 168%: delegation carries a real ~+6.6k-token premium on the +cheap tier for identical task completion — accepted because the frozen checker requires +delegate-or-script on a non-scriptable task and fable is the population of concern. Single +fixture. E5P mltok variance is high on fable (15k–65k baseline; 22k–47k candidate). + +## Transferable Lessons + +- Econ guards on cheap tiers should be absolute-token or scriptability-conditioned, not + ratios of a grind baseline — a ratio guard punishes correct delegation when the baseline + behavior is an efficient grind. (Promoted to [[eval-methodology-ladder]].) +- Account session limits truncate headless reps into plausible-looking behavior FAILs (and + PASSes); sweep every rep's final assistant message for the limit banner before counting. + (Promoted to [[running-autoresearch-skill-evals]].) +- Reserve twins authored before a criterion redesign can be silently unscoreable by the new + checker; audit checker constants against every planted fixture incident (not just run-set) + before the reserve run — fixable by id-keyed criteria without reading reserve files. +- Delegating reserve-twin authoring to a subagent (report = filenames only) keeps held-out + content out of the tuning session's context — cheap insurance when the same session will + run the wording loop. + +**Promotion check:** promoted lesson 1 to [[eval-methodology-ladder]] and lesson 2 to +[[running-autoresearch-skill-evals]] (both updated 2026-07-08); lessons 3–4 not yet +recurring. + +## Deployment and Evolution + +**Good-Enough Gate** — shipped 2026-07-08 (ORCHESTRATION.md v3 in cc-os + refreshed cache): +negatives clean on both grids, batching generalizes to the reserve twin on both tiers, H3 +met on the pre-registered axis. Residual known gaps at ship time: E1P sonnet delegate-at-all +(fixture-scale, not wording), fable downgrade-flag ~1/4 judge-boundary flicker (os-adr W3 +shape — characterize in production, don't lab-chase). + +**Proposed Changes (Loop Inputs)** — none; candidate-1 shipped verbatim (five changes: +cost-asymmetry + tier-conditional threshold opener, symmetric delegate/work-directly trigger +lists, batching rule ~5–8 items/spawn + SendMessage reuse, async rule, Workflow effort +dial). Diff against v2 is commit-visible in cc-os. + +**Hardening Path / Next Measurement** — production IRL re-audit of ~5 real Fable sessions +scheduled ~2026-07-22 per [[eval-methodology-irl-feedback-loop]]: compare main-loop token +share vs the 2026-07-08 mini-audit baseline (now measurable exactly via subagent +transcripts); promote new miss shapes into eval scenarios. E1P needs a larger fixture before +its sonnet cell is meaningful. + +## Related + +- [[os-orchestration-eval-baseline-grid-results]] — the 2026-07-06 E1–E3 baseline this loop builds on +- [[orchestration-prompting-claude-5-era]] — the model-page guidance the v3 wording encodes +- [[eval-methodology-ladder]] / [[running-autoresearch-skill-evals]] — discipline followed + lessons promoted +- [[claude-code-subagent-transcripts-and-token-accounting]] — the instrumentation discovery behind the econ axes