SecondBrain/eval-methodology-irl-feedback-loop.md

---
type: reference
subtype: pattern/framework
title: Production validation via session auditing — turning silent misses into eval scenarios
summary: A pattern for treating post-rollout observation as a deliberate, recurring backlog item: periodically audit real sessions in onboarded projects, log should-have-triggered vs did-trigger patterns, and promote recurring misses into new eval scenarios.
tags:
  - type/reference
  - domain/llm-evaluation
  - convention/eval-design
scope: global
last_updated: 2026-07-06
date: 2026-07-06
related:
  - eval-methodology-ladder
  - running-autoresearch-skill-evals
  - os-adr-eval-b-grid-results-and-observations
source: cc-os
---

# Production validation via session auditing

## Purpose

Evaluations measure behavior in controlled scenarios. Production measures behavior in messy reality — different codebases, different decision patterns, different user workflows. Evals can pass while production silently fails. This pattern establishes a deliberate backlog item for discovering and closing the gap.

Why is "observation" not passive? Because **misses are silent.** If a model fails to record an architecture decision, nothing visibly breaks — decision history drifts, but the project works. A broken error message fails loud (user complains); a missed decision record fails silent (history corrupts, unnoticed for months). Passive hope ("users will tell us") doesn't work for silent failures. You must audit deliberately.

## Core Principles

**1. Silent misses require active instrumentation.** A feature that fails loud (raises an error) gives feedback immediately. A feature that fails silent (omits a step, forgets to record) requires sampling and analysis to surface. Post-rollout observation must be structured: pick a sample of sessions in real projects, judge each against a clear criterion ("should-have-triggered, did-trigger?"), and log the patterns.

**2. The audit is not about blame — it's about scenario authoring.** When a session shows "should have triggered, didn't," the task is not "why did the model fail?" (blame) but "what aspect of this scenario isn't covered in our current eval?" (design). Each recurring miss shape becomes a new scenario for a follow-up eval.

**3. The feedback loop closes when new scenarios enter the eval.** A miss discovered in audit that never makes it into a future eval is a missed opportunity. The pattern succeeds when production findings become new Eval Level 2 / 3 scenarios (see [[eval-methodology-ladder]]); the loop closes when improved wording / architecture based on those scenarios gets deployed back to the same onboarded projects.

**4. Cadence matters.** Auditing is not one-time; it's recurring. After the first 1–2 weeks in a onboarded project (when user activity is highest and model interactions are freshest), run the first audit. Then audit every 2 weeks or per-project-phase depending on decision velocity.

## Decision Framework

| When | What to audit |
|---|---|
| First 1–2 weeks after rollout to a new project | Sample 10–20 sessions (or all sessions if the project has <20 in the period). Look for false negatives ("should have triggered, didn't") and false positives ("over-triggered, shouldn't have"). Establish baseline miss patterns. |
| After wording/architecture tuning from a prior audit | Sample 5–10 sessions to check whether the changes reduced the miss rate. If misses persist in new shapes, recurse to the "analyze" step. |
| Steady-state (3+ projects live, stable wording) | Audit one project per cycle (round-robin) with sample size 5. Track whether miss patterns are consistent across projects (fixture-specific vs universal). |

**Session sampling strategy:**
- **Random or systematic:** Pick every Nth session from the project's audit log, or randomly sample a date range. Avoid cherry-picking (audit shows surprises; cherry-pick hides them).
- **Scope:** A "session" is one Claude Code context window. Audit the entire conversation (full tool_use transcript), not just successes.
- **Sample size:** 10–20 sessions for the first audit (establishes baseline and catches ~70% of miss patterns with reasonable rarity filtering). 5 sessions for follow-ups (faster iteration, assuming earlier audits already surfaced the common patterns).

**Judging should-have-triggered:**
- Be explicit about the criterion. E.g., "should consult `/os-adr:find` if: (a) user is making or modifying an architecture-level choice, (b) the choice isn't totally novel to the codebase, and (c) the choice is documented anywhere in the ADR system." Write this down before auditing so raters don't drift.
- A "should-have-triggered" miss is when criterion (a)–(c) are all met, but the model didn't consult.
- An "over-triggered" miss is when the model consulted when only (a) was true but (b)–(c) were false (e.g., a decision so novel that no prior ADR applies).
- A "did-trigger, correct outcome" is not a miss — it's a hit.

## Patterns

### Pattern: Audit report template

```
**Project:** [name]
**Audit date:** [YYYY-MM-DD]
**Sessions sampled:** [count and date range]
**Auditor:** [who]

**Baseline metrics:**
- Total sessions sampled: [N]
- Sessions with relevant decisions: [N] (filtered: only count sessions where the criterion applies)
- Correct consultation rate: [N/M]
- Should-have-triggered, didn't: [N] (list scenarios)
- Over-triggered (false positive): [N] (list scenarios)
- Judge call / ambiguous: [N] (borderline cases where the criterion is unclear)

**Miss patterns identified:**
1. [Pattern name]: [description, example]
2. [Pattern name]: [description, example]
...

**New scenarios for next eval:**
- [Miss pattern 1] → Level [2/3] scenario: positive case [X], negative case [Y]
- [Miss pattern 2] → Level [2/3] scenario: positive case [X], negative case [Y]

**Recommendation:**
- Severity: [low/medium/high] (how often does this miss occur? how bad is the consequence?)
- Next step: [add to Eval Level N backlog / reword in CLAUDE.md / architecture change needed / defer]
```

### Pattern: Audit backlog and prioritization

Create a simple backlog of audits as a recurring task:
1. Audit newly onboarded projects after 1–2 weeks.
2. Track which projects have been audited and when.
3. Round-robin steady-state audits (one project per cycle).
4. Prioritize re-auditing projects that showed high miss rates in prior cycles.

Prioritize misses by:
- **Severity:** How often does the miss occur? (rare fluke vs systematic gap)
- **Consequence:** What breaks if the miss goes undetected? (silent corruption vs visible error)
- **Design feasibility:** Can this be fixed by wording, or does it require architecture change?

High-severity, high-consequence misses with feasible fixes become new Eval Level 2 scenarios immediately.

### Pattern: Closing the loop — from audit to eval to deployment

1. **Audit surfaced a miss pattern** → log it in the audit report.
2. **New scenarios authored** → author Level 2 or 3 scenarios that reproduce the miss (positive: the correct case; negative: the false-premise case that was missed).
3. **Eval run against the scenarios** → confirm the miss (model fails the scenario in the same way it failed in production).
4. **Wording/architecture tuning** → iterate based on the new scenarios (in a controlled eval, not directly in production).
5. **Deploy the tuning** → ship the improved wording to the onboarded projects.
6. **Follow-up audit** → run a quick re-audit on the project where the miss was found to confirm the fix worked.

This cycle is the **feedback loop closure**. Projects become co-authors of the eval through their production experiences.

## Anti-Patterns

- **Auditing without a clear criterion.** "Judge whether the model should have consulted the ADR system" is too vague. Write the exact conditions and stick to them across audits.
- **One-time audit and moving on.** Post-rollout observation that doesn't recur is passive hope wearing a label. Recurring audits catch drift over time and provide feedback for the next eval cycle.
- **Auditing successes only.** If you only sample sessions that went well, you'll miss the silent failures. Audit randomly, or weight toward sessions with *decisions* (the feature is only relevant in decision-heavy work).
- **Misses that never become scenarios.** If a production miss is identified but never authoring as an eval scenario, it's context-specific feedback with no leverage. High-severity misses must enter the eval backlog; otherwise, the feature never improves.
- **Conflating audit findings with blame.** "The model failed because the wording is bad" is defensible. "The model failed because it's stupid" is not actionable and demoralizes. Audits are design feedback, not model critiques.

## Known Limitations

Auditing requires access to real-session transcripts, which may have privacy implications if the projects involve clients or sensitive work. Ensure audit access is governed by the same policies as the vault (e.g., local-only, encrypted).

Auditing is labor-intensive (10–20 sessions × ~5–10 min per session judgment = 50–200 min per audit). Scaling to many projects requires either automation (e.g., heuristic pre-filtering for decision-relevant sessions) or delegating audit work to multiple raters (and managing inter-rater consistency). A third path is to run audits less frequently (per-quarter instead of bi-weekly) for mature deployments.

The pattern assumes the model's behavior is the variable under test. If the "miss" is actually a user workflow issue (user didn't provide enough context for a decision to be visible), that's not a model failure — it's a feature-discovery or onboarding gap. Keep that distinction clear in audit reports.

## Related

- [[eval-methodology-hub]] — navigation hub for all evaluation-methodology notes
- [[eval-methodology-ladder]] — how audits feed into next-level scenarios for the eval progression
- [[running-autoresearch-skill-evals]] — how to run evals that incorporate audit-sourced scenarios
- [[os-adr-eval-b-grid-results-and-observations]] — the baseline eval before production rollout