vault: session notes 2026-07-10
This commit is contained in:
parent
12e28cbb0c
commit
5adb3a306c
|
|
@ -0,0 +1,29 @@
|
|||
---
|
||||
summary: "Repeatable method for auditing Claude Code session transcripts with cheap subagents — deterministic extraction, batch fan-out to sonnet analysts, synthesis into coach/consultant findings."
|
||||
tags:
|
||||
- type/howto
|
||||
- domain/ai-workflow
|
||||
- tool/claude-code
|
||||
scope: global
|
||||
type: howto
|
||||
source: transcript audit 2026-07-09
|
||||
date: 2026-07-09
|
||||
last_updated: 2026-07-09
|
||||
---
|
||||
|
||||
# How to Audit AI Session Transcripts
|
||||
|
||||
Run on demand or schedule weekly (`/schedule`). Produces coach (work faster, less effort) + consultant (process/tooling levers) findings. First run: 2026-07-09; results in [[ai-workflow-audit-findings-jared]].
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Deterministic extraction (no LLM).** Transcripts live in `~/.claude/projects/<slug>/*.jsonl`. With `jq`, keep rows where `.type=="user" and .isMeta != true`, join text content, drop `<system-reminder>`/`<command-name>`/interrupt rows, cap text at 2000 chars, emit `{p: project, s: session, ts, len, text}`. Exclude scratchpad/eval/tmp project dirs. Filter `-mtime -45`.
|
||||
2. **Know the contamination trap.** 50-70% of extracted "user" rows are Claude-authored subagent-dispatch prompts — their session IDs start with `agent-`, and single-turn sessions are mostly these. Instruct analysts to exclude `agent-*` sessions from user-behavior dimensions (use them only for orchestration-efficiency findings). Interactive baseline = sessions with ≥3 user turns.
|
||||
3. **Compute deterministic stats first**: message-length distribution (median vs mean exposes paste-skew), turns per session, % terse messages, per-project volume. Anchors the coaching in numbers, not vibes.
|
||||
4. **Batch ~600-800KB per analyst** (split big projects, group small ones) and fan out parallel `sonnet` general-purpose agents with a shared rubric file covering: over-specification, effort waste, correction loops (+root cause), repeated instructions (persist candidates), what works, working-style profile, deterministic-tool opportunities. Require verbatim quotes and counts.
|
||||
5. **Subagents cannot write report files** (harness policy blocks report/markdown writes from subagents) — have them return findings as text; the parent saves them.
|
||||
6. **Synthesize** in the main loop: weight for the agent-prompt contamination, merge repeated-instruction counts across batches, output report + update the findings vault note.
|
||||
|
||||
## Cost profile
|
||||
|
||||
First run: 6 sonnet analysts, ~650K subagent tokens total, ~2.5 min wall time each, fully parallel.
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
---
|
||||
summary: "Cross-project audit findings on how Jared works with Claude Code — working-style profile, repeated instructions worth persisting, and deterministic-tool opportunities (audited 2026-07-09, 45 days of transcripts)."
|
||||
tags:
|
||||
- type/reference
|
||||
- domain/ai-workflow
|
||||
- tool/claude-code
|
||||
scope: global
|
||||
type: reference
|
||||
source: transcript audit 2026-07-09
|
||||
date: 2026-07-09
|
||||
last_updated: 2026-07-09
|
||||
---
|
||||
|
||||
# AI Workflow Audit Findings (Jared × Claude Code)
|
||||
|
||||
Full report: `~/Desktop/ai-workflow-audit-2026-07-09.md`. Method: [[audit-ai-session-transcripts]]. Corpus: 3,058 extracted messages / 45 days / 15+ projects; ~600-700 genuinely human-authored across 157 interactive sessions after filtering Claude-authored subagent-dispatch prompts (50-70% of raw rows).
|
||||
|
||||
## Working-style profile
|
||||
|
||||
Orchestrator-architect: makes every architectural/business call personally, delegates nearly all execution to tightly scoped subagents. Median interactive message 413 chars; 20-27% of direct messages are terse approvals ("proceed", "commit this") — the propose→approve loop is the default and most effective mode. High quality bar on maintainability and client-facing correctness; low trust in single-agent claims — personal verification built into nearly every mutating task, and it repeatedly caught real agent errors. Risk-escalates agent scope (read-only recon → guarded writes → verify). Plans extensively for architectural work, dives in tactically. Ritualizes session close. Designs around ADHD: pull-based dashboards over push digests. Expects and accepts pushback. Prefers Ruby > Python > bash for scripts; Sandi Metz OOP; TDD; lowercase-kebab filenames.
|
||||
|
||||
## Where effort is wasted (coach findings)
|
||||
|
||||
1. **Kickoff re-briefing**: house conventions (TDD, Sandi Metz, vertical slices, delegation defaults, "audit defaults before customizing") restated at nearly every project/gem kickoff — should be one vault pointer.
|
||||
2. **Safety preambles retyped per dispatch**: "read-only, do NOT change anything" appeared near-verbatim 15+ times; "never echo the secret", "release.sh needs explicit authorization" similar — template candidates.
|
||||
3. **Session-close ritual typed 20+ times** (commit + docs + todos + handoff/vault) — top candidate for a `/wrap` skill or SessionEnd hook.
|
||||
4. Gotchas re-derived on second encounter (wp-cli quoting, SSH-key recovery, Docker PATH) — rule: any environment surprise costing >3 turns gets a runbook/vault note before session end.
|
||||
5. Deterministic tools built at step six, not step one (release.sh after ~6h of manual loops; verify script only after an incident).
|
||||
6. Client-side double-submit bug: ~8% duplicate messages in one batch.
|
||||
|
||||
## Repeated-instruction persistence candidates (by frequency)
|
||||
|
||||
- Session-close ritual (20+) → `/wrap` skill / SessionEnd hook
|
||||
- Read-only/guarded-write safety preamble (15+) → dispatch template or CLAUDE.md rule
|
||||
- "Dispatch subagents to..." (12+) → global orchestration default (partly exists)
|
||||
- "Commit this / commit and push" (9+) → wrap into session-close
|
||||
- TDD/Sandi Metz/gem-stack conventions (per kickoff) → conventions note + gem scaffold generator
|
||||
- Plan-before-implement gate; staging-first deploys; destructive-op pre-flight → hooks/rules, not prose
|
||||
|
||||
## Deterministic-tool opportunities
|
||||
|
||||
Task-checkbox flipping (~10+ LLM dispatches for pure mechanics); Proxmox VM lifecycle script (5× re-dispatched); SSH-hardening drop-in playbook; Docker image lift-and-shift (3×); dns-preflight; ACF write wrapper; doc-citation drift check in CI; use installed os-doc-hygiene instead of ad-hoc staleness sweeps (re-invented 5+ times).
|
||||
|
||||
## Keep doing
|
||||
|
||||
Propose→approve with numbered options; pointing at files/URLs instead of pasting; phased risk escalation with personal verification; empirical dispute resolution (frozen-grid evals); scope self-correction before code exists.
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
type: reference
|
||||
subtype: api-integration
|
||||
title: "Vaultwarden vs Bitwarden Cloud — permission and feature parity gotchas"
|
||||
summary: Verified differences between self-hosted Vaultwarden and Bitwarden Cloud that break security designs copied from Bitwarden docs — Edit permission includes delete, no deletion-restriction org policy, no Secrets Manager, trash never auto-purges by default.
|
||||
summary: Verified differences between self-hosted Vaultwarden and Bitwarden Cloud that break security designs copied from Bitwarden docs — Edit permission includes delete, no deletion-restriction org policy, no Secrets Manager, trash never auto-purges by default — plus org bot-onboarding gotchas (accept ≠ confirm, collection assignment can silently not persist).
|
||||
tags:
|
||||
- type/reference
|
||||
- tool/vaultwarden
|
||||
|
|
@ -38,5 +38,21 @@ issue #6269):
|
|||
- **`bw list items --search` doesn't match custom fields** — filter client-side on parsed JSON.
|
||||
- Maintainer (BlackDex) states Vaultwarden "does not support the full range of RBAC/GBAC/CBAC".
|
||||
|
||||
## Org member (bot) onboarding gotchas (verified 2026-07-09, credvault Phase 0)
|
||||
|
||||
- **Accept ≠ confirm.** A member who accepts an invite still sees NO organization via
|
||||
`bw list organizations` until an org admin explicitly clicks **Confirm** on the member
|
||||
(Members page → three-dot menu). Confirm is the step that grants the org key; editing the
|
||||
member's role/collections is not it.
|
||||
- **Collection assignment can silently fail to persist** — after the admin sets a collection
|
||||
in Edit member → Collections, `bw list collections` (bot side) can still return `[]` after
|
||||
sync. Verify the collection badge shows on the Members list row; re-open the dialog and
|
||||
re-save if not.
|
||||
- **`bw sync -f` after any server-side membership change** — login/unlock state alone won't
|
||||
refresh org/collection visibility on the client.
|
||||
- **Password history verified working** on Vaultwarden: `bw edit` with a changed login
|
||||
password preserves the old one in `passwordHistory` with `lastUsedDate`. Custom `fields`
|
||||
round-trip intact through `bw list items` JSON.
|
||||
|
||||
Rule of thumb: any "Bitwarden supports X" claim from bitwarden.com/help must be re-verified
|
||||
against the deployed Vaultwarden version before it becomes a security assumption.
|
||||
|
|
|
|||
Loading…
Reference in New Issue