jared
/
SecondBrain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SecondBrain/howto/audit-ai-session-transcript...

2.4 KiB
Raw Blame History

summary tags scope type source date last_updated
Repeatable method for auditing Claude Code session transcripts with cheap subagents — deterministic extraction, batch fan-out to sonnet analysts, synthesis into coach/consultant findings.
type/howto
domain/ai-workflow
tool/claude-code
global howto transcript audit 2026-07-09 2026-07-09 2026-07-09

How to Audit AI Session Transcripts

Run on demand or schedule weekly (/schedule). Produces coach (work faster, less effort) + consultant (process/tooling levers) findings. First run: 2026-07-09; results in ai-workflow-audit-findings-jared.

Steps

  1. Deterministic extraction (no LLM). Transcripts live in ~/.claude/projects/<slug>/*.jsonl. With jq, keep rows where .type=="user" and .isMeta != true, join text content, drop <system-reminder>/<command-name>/interrupt rows, cap text at 2000 chars, emit {p: project, s: session, ts, len, text}. Exclude scratchpad/eval/tmp project dirs. Filter -mtime -45.
  2. Know the contamination trap. 50-70% of extracted "user" rows are Claude-authored subagent-dispatch prompts — their session IDs start with agent-, and single-turn sessions are mostly these. Instruct analysts to exclude agent-* sessions from user-behavior dimensions (use them only for orchestration-efficiency findings). Interactive baseline = sessions with ≥3 user turns.
  3. Compute deterministic stats first: message-length distribution (median vs mean exposes paste-skew), turns per session, % terse messages, per-project volume. Anchors the coaching in numbers, not vibes.
  4. Batch ~600-800KB per analyst (split big projects, group small ones) and fan out parallel sonnet general-purpose agents with a shared rubric file covering: over-specification, effort waste, correction loops (+root cause), repeated instructions (persist candidates), what works, working-style profile, deterministic-tool opportunities. Require verbatim quotes and counts.
  5. Subagents cannot write report files (harness policy blocks report/markdown writes from subagents) — have them return findings as text; the parent saves them.
  6. Synthesize in the main loop: weight for the agent-prompt contamination, merge repeated-instruction counts across batches, output report + update the findings vault note.

Cost profile

First run: 6 sonnet analysts, ~650K subagent tokens total, ~2.5 min wall time each, fully parallel.