diff --git a/howto/audit-ai-session-transcripts.md b/howto/audit-ai-session-transcripts.md new file mode 100644 index 0000000..394f3dd --- /dev/null +++ b/howto/audit-ai-session-transcripts.md @@ -0,0 +1,29 @@ +--- +summary: "Repeatable method for auditing Claude Code session transcripts with cheap subagents — deterministic extraction, batch fan-out to sonnet analysts, synthesis into coach/consultant findings." +tags: + - type/howto + - domain/ai-workflow + - tool/claude-code +scope: global +type: howto +source: transcript audit 2026-07-09 +date: 2026-07-09 +last_updated: 2026-07-09 +--- + +# How to Audit AI Session Transcripts + +Run on demand or schedule weekly (`/schedule`). Produces coach (work faster, less effort) + consultant (process/tooling levers) findings. First run: 2026-07-09; results in [[ai-workflow-audit-findings-jared]]. + +## Steps + +1. **Deterministic extraction (no LLM).** Transcripts live in `~/.claude/projects//*.jsonl`. With `jq`, keep rows where `.type=="user" and .isMeta != true`, join text content, drop ``/``/interrupt rows, cap text at 2000 chars, emit `{p: project, s: session, ts, len, text}`. Exclude scratchpad/eval/tmp project dirs. Filter `-mtime -45`. +2. **Know the contamination trap.** 50-70% of extracted "user" rows are Claude-authored subagent-dispatch prompts — their session IDs start with `agent-`, and single-turn sessions are mostly these. Instruct analysts to exclude `agent-*` sessions from user-behavior dimensions (use them only for orchestration-efficiency findings). Interactive baseline = sessions with ≥3 user turns. +3. **Compute deterministic stats first**: message-length distribution (median vs mean exposes paste-skew), turns per session, % terse messages, per-project volume. Anchors the coaching in numbers, not vibes. +4. **Batch ~600-800KB per analyst** (split big projects, group small ones) and fan out parallel `sonnet` general-purpose agents with a shared rubric file covering: over-specification, effort waste, correction loops (+root cause), repeated instructions (persist candidates), what works, working-style profile, deterministic-tool opportunities. Require verbatim quotes and counts. +5. **Subagents cannot write report files** (harness policy blocks report/markdown writes from subagents) — have them return findings as text; the parent saves them. +6. **Synthesize** in the main loop: weight for the agent-prompt contamination, merge repeated-instruction counts across batches, output report + update the findings vault note. + +## Cost profile + +First run: 6 sonnet analysts, ~650K subagent tokens total, ~2.5 min wall time each, fully parallel. diff --git a/reference/ai-workflow-audit-findings-jared.md b/reference/ai-workflow-audit-findings-jared.md new file mode 100644 index 0000000..185172a --- /dev/null +++ b/reference/ai-workflow-audit-findings-jared.md @@ -0,0 +1,46 @@ +--- +summary: "Cross-project audit findings on how Jared works with Claude Code — working-style profile, repeated instructions worth persisting, and deterministic-tool opportunities (audited 2026-07-09, 45 days of transcripts)." +tags: + - type/reference + - domain/ai-workflow + - tool/claude-code +scope: global +type: reference +source: transcript audit 2026-07-09 +date: 2026-07-09 +last_updated: 2026-07-09 +--- + +# AI Workflow Audit Findings (Jared × Claude Code) + +Full report: `~/Desktop/ai-workflow-audit-2026-07-09.md`. Method: [[audit-ai-session-transcripts]]. Corpus: 3,058 extracted messages / 45 days / 15+ projects; ~600-700 genuinely human-authored across 157 interactive sessions after filtering Claude-authored subagent-dispatch prompts (50-70% of raw rows). + +## Working-style profile + +Orchestrator-architect: makes every architectural/business call personally, delegates nearly all execution to tightly scoped subagents. Median interactive message 413 chars; 20-27% of direct messages are terse approvals ("proceed", "commit this") — the propose→approve loop is the default and most effective mode. High quality bar on maintainability and client-facing correctness; low trust in single-agent claims — personal verification built into nearly every mutating task, and it repeatedly caught real agent errors. Risk-escalates agent scope (read-only recon → guarded writes → verify). Plans extensively for architectural work, dives in tactically. Ritualizes session close. Designs around ADHD: pull-based dashboards over push digests. Expects and accepts pushback. Prefers Ruby > Python > bash for scripts; Sandi Metz OOP; TDD; lowercase-kebab filenames. + +## Where effort is wasted (coach findings) + +1. **Kickoff re-briefing**: house conventions (TDD, Sandi Metz, vertical slices, delegation defaults, "audit defaults before customizing") restated at nearly every project/gem kickoff — should be one vault pointer. +2. **Safety preambles retyped per dispatch**: "read-only, do NOT change anything" appeared near-verbatim 15+ times; "never echo the secret", "release.sh needs explicit authorization" similar — template candidates. +3. **Session-close ritual typed 20+ times** (commit + docs + todos + handoff/vault) — top candidate for a `/wrap` skill or SessionEnd hook. +4. Gotchas re-derived on second encounter (wp-cli quoting, SSH-key recovery, Docker PATH) — rule: any environment surprise costing >3 turns gets a runbook/vault note before session end. +5. Deterministic tools built at step six, not step one (release.sh after ~6h of manual loops; verify script only after an incident). +6. Client-side double-submit bug: ~8% duplicate messages in one batch. + +## Repeated-instruction persistence candidates (by frequency) + +- Session-close ritual (20+) → `/wrap` skill / SessionEnd hook +- Read-only/guarded-write safety preamble (15+) → dispatch template or CLAUDE.md rule +- "Dispatch subagents to..." (12+) → global orchestration default (partly exists) +- "Commit this / commit and push" (9+) → wrap into session-close +- TDD/Sandi Metz/gem-stack conventions (per kickoff) → conventions note + gem scaffold generator +- Plan-before-implement gate; staging-first deploys; destructive-op pre-flight → hooks/rules, not prose + +## Deterministic-tool opportunities + +Task-checkbox flipping (~10+ LLM dispatches for pure mechanics); Proxmox VM lifecycle script (5× re-dispatched); SSH-hardening drop-in playbook; Docker image lift-and-shift (3×); dns-preflight; ACF write wrapper; doc-citation drift check in CI; use installed os-doc-hygiene instead of ad-hoc staleness sweeps (re-invented 5+ times). + +## Keep doing + +Propose→approve with numbered options; pointing at files/URLs instead of pasting; phased risk escalation with personal verification; empirical dispute resolution (frozen-grid evals); scope self-correction before code exists. diff --git a/reference/vaultwarden-permission-parity-gotchas.md b/reference/vaultwarden-permission-parity-gotchas.md index c94aa4e..3a73ccb 100644 --- a/reference/vaultwarden-permission-parity-gotchas.md +++ b/reference/vaultwarden-permission-parity-gotchas.md @@ -2,7 +2,7 @@ type: reference subtype: api-integration title: "Vaultwarden vs Bitwarden Cloud — permission and feature parity gotchas" -summary: Verified differences between self-hosted Vaultwarden and Bitwarden Cloud that break security designs copied from Bitwarden docs — Edit permission includes delete, no deletion-restriction org policy, no Secrets Manager, trash never auto-purges by default. +summary: Verified differences between self-hosted Vaultwarden and Bitwarden Cloud that break security designs copied from Bitwarden docs — Edit permission includes delete, no deletion-restriction org policy, no Secrets Manager, trash never auto-purges by default — plus org bot-onboarding gotchas (accept ≠ confirm, collection assignment can silently not persist). tags: - type/reference - tool/vaultwarden @@ -38,5 +38,21 @@ issue #6269): - **`bw list items --search` doesn't match custom fields** — filter client-side on parsed JSON. - Maintainer (BlackDex) states Vaultwarden "does not support the full range of RBAC/GBAC/CBAC". +## Org member (bot) onboarding gotchas (verified 2026-07-09, credvault Phase 0) + +- **Accept ≠ confirm.** A member who accepts an invite still sees NO organization via + `bw list organizations` until an org admin explicitly clicks **Confirm** on the member + (Members page → three-dot menu). Confirm is the step that grants the org key; editing the + member's role/collections is not it. +- **Collection assignment can silently fail to persist** — after the admin sets a collection + in Edit member → Collections, `bw list collections` (bot side) can still return `[]` after + sync. Verify the collection badge shows on the Members list row; re-open the dialog and + re-save if not. +- **`bw sync -f` after any server-side membership change** — login/unlock state alone won't + refresh org/collection visibility on the client. +- **Password history verified working** on Vaultwarden: `bw edit` with a changed login + password preserves the old one in `passwordHistory` with `lastUsedDate`. Custom `fields` + round-trip intact through `bw list items` JSON. + Rule of thumb: any "Bitwarden supports X" claim from bitwarden.com/help must be re-verified against the deployed Vaultwarden version before it becomes a security assumption.