vault: session notes 2026-07-04

2026-07-03-os-adr-eval-b-grid-results-and-observations.md

@@ -1,5 +1,6 @@
 ---
-summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures — read before designing any follow-up eval on unprompted-consultation behavior.
+summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures (confirmed by 2026-07-04 re-run) — read before designing any follow-up eval on unprompted-consultation behavior.
+last_updated: 2026-07-04
 tags:
   - scope/global
   - type/reference
@@ -70,8 +71,9 @@ failed for haiku.
 Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing
 SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands
 (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two
-behavioral failures (W3, R1).** 15/16 cells reproduced; the 16th (W2/sonnet) was a harness error
-(no transcript), re-run separately.
+behavioral failures (W3, R1).** All 16 cells reproduced (one cell, W2/sonnet, needed a clean
+re-run after a harness error — no transcript captured on the first attempt — and PASSed on
+re-run, matching baseline).
 
 New information beyond confirmation:
 

journal/2026-07-04.md

@@ -457,3 +457,10 @@ tags: [scope/global, type/log]
 **Reason:** prompt_input_exit
 **Vault notes touched:**
 (none)
+
+## Session — 2026-07-04T17:20:34Z
+
+**Project:** /home/jared/dev/cc-os
+**Reason:** other
+**Vault notes touched:**
+(none)

os-adr-eval-b-wording-experiment-hypotheses.md

@@ -0,0 +1,80 @@
+---
+summary: Hypotheses for the os-adr Eval B wording experiment (trigger-conditioned phrasing, packaging channel, two-step trigger chain) and the explicit map from grid outcomes to hypothesis verdicts — the tracking pattern any skill-wording eval should copy.
+tags:
+  - scope/global
+  - type/reference
+  - project/cc-os
+  - tool/os-adr
+  - tool/autoresearch
+  - domain/llm-evaluation
+source: cc-os
+date: 2026-07-04
+last_updated: 2026-07-04
+related:
+  - os-adr-eval-b-grid-results-and-observations
+  - running-autoresearch-skill-evals
+---
+
+# os-adr Eval B wording experiment — hypotheses and result mapping
+
+Design note for the 2026-07-04 `/autoresearch` wording loop against the Eval B baseline
+(haiku 0/8, sonnet 5/8 failing W3 + R1 — see
+[[os-adr-eval-b-grid-results-and-observations]]). Procedure discipline lives in
+[[running-autoresearch-skill-evals]]; this note holds *what we believe and how we'll know*.
+
+## Framing
+
+Eval A (prompted) passes 100% on both tiers → the instructions are **followable**. Eval B
+(unprompted) fails → the problem is **situation recognition**: the model is told *what*
+exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but
+never classifies its *current moment* as ADR-relevant. Analogy that framed it: telling
+someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."
+
+## Hypotheses
+
+**H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about
+to make an architecture-level choice, or change an approach that already exists in the
+codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever
+channel they appear. The model needs recognition cues to pattern-match its current task
+against, not facts to spontaneously recall.
+
+**H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's
+present at decision time. Candidate channels: SessionStart hook note (seen once, may fade),
+project CLAUDE.md line (in context every session), skill `description` frontmatter
+(ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be
+redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation.
+
+**H3 — Two-step trigger chain (instruction placement):** Each condition must reference only
+what the model can observe *at that point*. "Reversing a prior decision" is unknowable
+before consulting — so step 1 triggers on locally observable conditions (making/changing an
+architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted
+ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in
+the find/create skill bodies (visible when find's results are on screen), not the
+session-start note (long out of attention by then).
+
+## How grid outcomes map to hypotheses
+
+| Outcome | Verdict |
+|---|---|
+| Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for the consultation trigger |
+| Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies | Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it |
+| Ablation: CLAUDE.md-only arm holds sonnet 8/8 without the hook note | Confirms H2 sub-hypothesis (hook redundant); keep hook only if its removal regresses |
+| Ablation: hook-only arm regresses vs CLAUDE.md-only | Channel strength ordering: persistent-context > session-start injection |
+| Haiku moves off 0/8 with the winning sonnet packaging | H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover |
+| Haiku stays 0/8 after ~3 iterations of the sonnet-winning wording | Capability floor for unprompted triggering at haiku tier — record it and stop iterating |
+| W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains |
+| Any target cell moves on <2 of 3 reps | Noise, not signal — do not accept the change (variance demonstrated 2026-07-03/04) |
+
+## Meta-lesson (why this note exists)
+
+Track hypothesis → expected-outcome mapping **before** running a wording loop. It forces
+each grid cell to have a declared meaning, prevents post-hoc rationalization of noisy
+flips, and lets later evals zero in faster by reusing confirmed mechanisms (e.g., if H1
+confirms here, start the *next* plugin's unprompted-behavior wording at trigger-conditioned
+phrasing instead of rediscovering it). Pair this note's pattern with the procedure in
+[[running-autoresearch-skill-evals]].
+
+## Status
+
+- 2026-07-04: hypotheses locked, experiment starting (baseline commit cc-os `c645faa`).
+  Results to be appended here per iteration.

reference/bash-pipefail-silent-exit-gotcha.md

@@ -0,0 +1,59 @@
+---
+type: reference
+title: Bash pipefail silent-exit gotcha (grep in command substitution)
+summary: Under set -euo pipefail, VAR=$(grep ... | ...) silently kills the script when grep finds no match — and the debugging lesson that "works interactively, fails in cron" claims must be tested by re-running the full script under its own shell options.
+tags:
+  - scope/global
+  - type/reference
+  - domain/bash
+  - domain/debugging
+---
+
+# Bash pipefail silent-exit gotcha
+
+## The failure mode
+
+Under `set -euo pipefail`, a variable assignment via command substitution that
+contains a grep pipeline:
+
+```bash
+VAR=$(grep -oE 'pattern' "$FILE" 2>/dev/null | tail -1 | awk '{print $2}')
+```
+
+**silently terminates the script when grep finds no match:**
+
+1. No match → `grep` exits 1
+2. `pipefail` propagates that through the pipeline (`tail`/`awk` exiting 0 doesn't rescue it)
+3. The assignment's exit status is the substitution's exit status → nonzero
+4. `set -e` exits the script — with **zero error output** (`set -e` prints nothing,
+   and here grep's stderr was `/dev/null`'d anyway)
+
+**Symptom signature:** script halts silently mid-run, no error anywhere in any log,
+everything before the halt point succeeded.
+
+**Guard pattern:**
+
+```bash
+VAR=$(pipeline || true)   # || true inside the substitution rescues pipefail
+```
+
+Or delete the line if it's dead code (a value that can never match on that host).
+
+## The debugging lesson (bigger than the gotcha)
+
+Confirmed in production 2026-07-04: the ovh-prod `backup.sh` halted silently every
+night for ~2 months (2026-05-08 → 2026-07-04) and was **misattributed to a
+cron-vs-interactive environment difference** across multiple debugging sessions.
+Why the misattribution survived: each suspect command (`find -delete`, the notifier)
+was re-run individually over SSH and worked — but the grep pipeline was never
+re-tested **under the script's own shell options**. The failure had nothing to do
+with cron; it reproduced identically in an interactive shell the moment the exact
+pipeline was run under `set -euo pipefail`.
+
+**Rule:** when a script "works interactively but fails under cron/systemd", before
+blaming the environment, re-run the *full script* (or the exact failing region under
+the same `set -euo pipefail` flags) interactively. Testing commands in isolation
+strips the shell-option context that may be the actual killer. Cheap first check:
+`bash -c 'set -euo pipefail; <suspect line>; echo REACHED'`.
+
+Incident write-up: `~/servers/ovh-prod/docs/incidents.md`, 2026-07-04 entry.