30 lines
2.4 KiB
Markdown
30 lines
2.4 KiB
Markdown
|
---
|
||
|
|
summary: "Repeatable method for auditing Claude Code session transcripts with cheap subagents — deterministic extraction, batch fan-out to sonnet analysts, synthesis into coach/consultant findings."
|
||
|
|
tags:
|
||
|
|
- type/howto
|
||
|
|
- domain/ai-workflow
|
||
|
|
- tool/claude-code
|
||
|
|
scope: global
|
||
|
|
type: howto
|
||
|
|
source: transcript audit 2026-07-09
|
||
|
|
date: 2026-07-09
|
||
|
|
last_updated: 2026-07-09
|
||
|
|
---
|
||
|
|
|
||
|
|
# How to Audit AI Session Transcripts
|
||
|
|
|
||
|
|
Run on demand or schedule weekly (`/schedule`). Produces coach (work faster, less effort) + consultant (process/tooling levers) findings. First run: 2026-07-09; results in [[ai-workflow-audit-findings-jared]].
|
||
|
|
|
||
|
|
## Steps
|
||
|
|
|
||
|
|
1. **Deterministic extraction (no LLM).** Transcripts live in `~/.claude/projects/<slug>/*.jsonl`. With `jq`, keep rows where `.type=="user" and .isMeta != true`, join text content, drop `<system-reminder>`/`<command-name>`/interrupt rows, cap text at 2000 chars, emit `{p: project, s: session, ts, len, text}`. Exclude scratchpad/eval/tmp project dirs. Filter `-mtime -45`.
|
||
|
|
2. **Know the contamination trap.** 50-70% of extracted "user" rows are Claude-authored subagent-dispatch prompts — their session IDs start with `agent-`, and single-turn sessions are mostly these. Instruct analysts to exclude `agent-*` sessions from user-behavior dimensions (use them only for orchestration-efficiency findings). Interactive baseline = sessions with ≥3 user turns.
|
||
|
|
3. **Compute deterministic stats first**: message-length distribution (median vs mean exposes paste-skew), turns per session, % terse messages, per-project volume. Anchors the coaching in numbers, not vibes.
|
||
|
|
4. **Batch ~600-800KB per analyst** (split big projects, group small ones) and fan out parallel `sonnet` general-purpose agents with a shared rubric file covering: over-specification, effort waste, correction loops (+root cause), repeated instructions (persist candidates), what works, working-style profile, deterministic-tool opportunities. Require verbatim quotes and counts.
|
||
|
|
5. **Subagents cannot write report files** (harness policy blocks report/markdown writes from subagents) — have them return findings as text; the parent saves them.
|
||
|
|
6. **Synthesize** in the main loop: weight for the agent-prompt contamination, merge repeated-instruction counts across batches, output report + update the findings vault note.
|
||
|
|
|
||
|
|
## Cost profile
|
||
|
|
|
||
|
|
First run: 6 sonnet analysts, ~650K subagent tokens total, ~2.5 min wall time each, fully parallel.
|