8.0 KiB

Raw Blame History

System-prompt profiles: assessment & recommended approach

Last updated: 2026-07-08 · Status: assessment only, no build started

Companion to the WS4 orchestration-economics work. Goal: reduce per-session context cost and improve focus by controlling what reaches the model's context — the built-in system prompt, and (more importantly) the layers we add on top of it.

What is mechanically possible today

Verified against claude --help (current install) and https://code.claude.com/docs/en/agent-sdk/modifying-system-prompts.md (fetched 2026-07-08):

Mechanism	What it does	Risk
`--append-system-prompt[-file]`	Adds instructions after the full default prompt	None — nothing removed
`--system-prompt[-file]`	Fully replaces the built-in prompt for the session	High — loses tool guidance, safety rules, env context; you must re-provide them
`--exclude-dynamic-system-prompt-sections`	Moves per-machine sections (cwd, git status, env, memory paths) into first user message → cross-session cache reuse	Low; env context carries slightly less weight
`--tools <list>` / `--disallowedTools`	Restricts the built-in toolset (removes tool schemas from context)	Low
`--settings` / `--setting-sources user,project,local`	Controls which settings layers (and therefore CLAUDE.md files, enabled plugins, hooks) load	Low
Output styles (`~/.claude/output-styles/*.md`, `keep-coding-instructions: true	false`)	Replace or extend the coding-instruction section of the default prompt; persistent, reusable across projects
Plugin enable/disable per session (`--settings` overlay or profile-specific settings file)	Each enabled plugin's skills/agents/hooks descriptions are injected every session	Low — biggest immediate lever
Agent SDK (`systemPrompt: {type:'preset', append, excludeDynamicSections}` or custom string)	Same controls programmatically; SDK default is a minimal prompt unless the preset is requested	—

Deferred-tool loading (ToolSearch) already ships in current CC — most tool schemas are no longer paid up front. That reduced the value of "gut the system prompt" since much of the old bloat was tool definitions.

Measured baseline (2026-07-08, headless haiku `-p` runs, `--output-format json` usage)

Configuration	Context tokens	Delta
`--setting-sources "" --tools ""` (system prompt text only)	6,347	—
`--setting-sources ""` (prompt + full built-in tool schemas)	24,277	+17.9k = tool schemas
Default settings, empty dir (user stack: 17 plugins, global CLAUDE.md, hooks)	29,042	+4.8k = user stack
Full stack, cc-os dir	41,765	+12.7k = project layer (CLAUDE.md ~11k dominates)

Interpretation:

The system prompt text is only ~6.3k tokens. The ~24k harness floor is dominated by tool schemas (~18k) — so the bottom-up lever is --tools restriction per profile, not prompt rewriting. Full --system-prompt replacement can save at most ~6k and carries the fork-maintenance risk.
The 17-plugin user stack costs ~4.8k, not tens of k — cheaper than estimated.
The single largest single item we control is the cc-os CLAUDE.md (~11k).
An interactive session that reads ~80k after one exchange is ~42k fixed overhead plus ~38k of turn-1 work product (web fetches, file reads) — conversation content, not harness bloat. Caveat: these baselines are headless haiku; an interactive Fable session's skill/agent listings and model-specific sections may differ somewhat — worth one /context check interactively.

Key finding: our bloat is mostly self-inflicted, not Anthropic's

Measured 2026-07-08 for a cc-os session:

cc-os CLAUDE.md: ~43 KB ≈ ~11k tokens, injected into every session. Much of it is an append-only implementation-status changelog, not orientation the session needs.
17 enabled plugins — every skill description (~80 skills), agent type, and hook usage note is injected into every session regardless of relevance (creative-team, invoice-ninja, rails-ui-component, api-wrapper… in a cc-os design session).
memsearch SessionStart recall injection: ~13 KB this session.
Multiple SessionStart hooks (orchestration, os-vault, os-status) each add context.

The built-in system prompt is a static, cached prefix (cache reads are ~10% of input price after the first turn within the 5-min TTL). The layers above are the same cost class and they're the ones we control. The "bloat → hallucination" link is context rot: attention degrades as context grows, whatever the source. Cutting 20k tokens of our own always-on injection beats fighting Anthropic's prompt.

Pi reality check: pi.dev makes no speed/cost/hallucination claims on its site. Its lean-ness comes from being a different harness (no MCP, no subagents, no permission system, minimal prompt + extensions) — not from stripping Claude Code. Fully replacing CC's system prompt while keeping CC's harness means maintaining a fork of prompt content the model was tuned around, re-broken by every CC update.

Recommended approach: subtract-first modular profiles

A cyolo <flags> shell wrapper that composes existing mechanisms per objective/project. Profiles mostly subtract (plugins, tools, setting sources) and lightly add (one append-file per objective). Full system-prompt replacement is a later, eval-gated experiment on headless sessions only.

Phase 0 — measure. Run /context in 3–4 representative session types (cc-os design, client dev, brainstorm) and record the actual breakdown. No optimizing before baselining.
Phase 1 — diet (no new machinery).
- Split cc-os CLAUDE.md: keep orientation (~2–3k tokens), move the implementation-status changelog to a linked doc the AI reads on demand (progressive disclosure).
- Per-profile plugin enablement: a settings overlay per profile listing enabledPlugins; a brainstorm profile doesn't load rails-ui-component/api-wrapper/invoice-ninja.
Phase 2 — cyolo profile wrapper.
- Central evergreen component prompts in one source dir (e.g. ~/.claude/profiles/): dev.md, ruby.md, planning.md (objective-focused: starting point → process → definition of done), brainstorm.md.
- Wrapper resolves profile from flag (--dev, --planning) + project detection (pwd, Gemfile, etc.), then launches claude --append-system-prompt-file <composed> --settings <profile-settings> [--tools …].
- Objective profiles state the session's endpoint explicitly ("this session ends when the plan is written and shared; implementation belongs to a later session").
Phase 3 — eval-gated replacement experiment (optional). Arize-style loop, matching the autoresearch playbook: scenario set × {default prompt, dieted stack, profile stack, full-replacement lean prompt} × model tiers; score task success + tokens/turn + wrong-tool/hallucination proxies from transcripts. Only promote full replacement if it wins on outcome, not just token count. Read ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md first, per standing rule.

Bottom-up track note: to audit the real shipped prompt, capture a live request body via a local logging proxy (ANTHROPIC_BASE_URL pointed at a one-file pass-through that dumps the system and tools arrays) — don't rely on the model to self-report its prompt. Given the 6.3k measurement, audit it to find what a lean profile can safely drop, but expect the --tools list to be where the real savings are.

Open questions

Per-model profile variants: deferred (user agrees it's too granular now); the wording-loop evals already show tier-dependent phrasing needs, so revisit after profiles exist.
Whether --setting-sources interacts cleanly with symlinked local plugins + hook absolute paths in settings.json (hooks are wired by absolute path and would still fire).
Where profile settings overlays live so they don't fight bin/refresh-plugins.

8.0 KiB Raw Blame History Unescape Escape