From b3f8ea95267b8fea9ddefbe3bc25baa75e2d20a8 Mon Sep 17 00:00:00 2001 From: jared Date: Sat, 4 Jul 2026 14:04:51 -0400 Subject: [PATCH] vault: session notes 2026-07-04 --- ...dr-eval-b-grid-results-and-observations.md | 8 +- journal/2026-07-04.md | 7 ++ ...dr-eval-b-wording-experiment-hypotheses.md | 80 +++++++++++++++++++ reference/bash-pipefail-silent-exit-gotcha.md | 59 ++++++++++++++ 4 files changed, 151 insertions(+), 3 deletions(-) create mode 100644 os-adr-eval-b-wording-experiment-hypotheses.md create mode 100644 reference/bash-pipefail-silent-exit-gotcha.md diff --git a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md index 7ba35ea..4f99e22 100644 --- a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +++ b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md @@ -1,5 +1,6 @@ --- -summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures — read before designing any follow-up eval on unprompted-consultation behavior. +summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures (confirmed by 2026-07-04 re-run) — read before designing any follow-up eval on unprompted-consultation behavior. +last_updated: 2026-07-04 tags: - scope/global - type/reference @@ -70,8 +71,9 @@ failed for haiku. Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two -behavioral failures (W3, R1).** 15/16 cells reproduced; the 16th (W2/sonnet) was a harness error -(no transcript), re-run separately. +behavioral failures (W3, R1).** All 16 cells reproduced (one cell, W2/sonnet, needed a clean +re-run after a harness error — no transcript captured on the first attempt — and PASSed on +re-run, matching baseline). New information beyond confirmation: diff --git a/journal/2026-07-04.md b/journal/2026-07-04.md index 2b6cb98..9dde74f 100644 --- a/journal/2026-07-04.md +++ b/journal/2026-07-04.md @@ -457,3 +457,10 @@ tags: [scope/global, type/log] **Reason:** prompt_input_exit **Vault notes touched:** (none) + +## Session — 2026-07-04T17:20:34Z + +**Project:** /home/jared/dev/cc-os +**Reason:** other +**Vault notes touched:** +(none) diff --git a/os-adr-eval-b-wording-experiment-hypotheses.md b/os-adr-eval-b-wording-experiment-hypotheses.md new file mode 100644 index 0000000..4ba88ac --- /dev/null +++ b/os-adr-eval-b-wording-experiment-hypotheses.md @@ -0,0 +1,80 @@ +--- +summary: Hypotheses for the os-adr Eval B wording experiment (trigger-conditioned phrasing, packaging channel, two-step trigger chain) and the explicit map from grid outcomes to hypothesis verdicts — the tracking pattern any skill-wording eval should copy. +tags: + - scope/global + - type/reference + - project/cc-os + - tool/os-adr + - tool/autoresearch + - domain/llm-evaluation +source: cc-os +date: 2026-07-04 +last_updated: 2026-07-04 +related: + - os-adr-eval-b-grid-results-and-observations + - running-autoresearch-skill-evals +--- + +# os-adr Eval B wording experiment — hypotheses and result mapping + +Design note for the 2026-07-04 `/autoresearch` wording loop against the Eval B baseline +(haiku 0/8, sonnet 5/8 failing W3 + R1 — see +[[os-adr-eval-b-grid-results-and-observations]]). Procedure discipline lives in +[[running-autoresearch-skill-evals]]; this note holds *what we believe and how we'll know*. + +## Framing + +Eval A (prompted) passes 100% on both tiers → the instructions are **followable**. Eval B +(unprompted) fails → the problem is **situation recognition**: the model is told *what* +exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but +never classifies its *current moment* as ADR-relevant. Analogy that framed it: telling +someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car." + +## Hypotheses + +**H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about +to make an architecture-level choice, or change an approach that already exists in the +codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever +channel they appear. The model needs recognition cues to pattern-match its current task +against, not facts to spontaneously recall. + +**H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's +present at decision time. Candidate channels: SessionStart hook note (seen once, may fade), +project CLAUDE.md line (in context every session), skill `description` frontmatter +(ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be +redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation. + +**H3 — Two-step trigger chain (instruction placement):** Each condition must reference only +what the model can observe *at that point*. "Reversing a prior decision" is unknowable +before consulting — so step 1 triggers on locally observable conditions (making/changing an +architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted +ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in +the find/create skill bodies (visible when find's results are on screen), not the +session-start note (long out of attention by then). + +## How grid outcomes map to hypotheses + +| Outcome | Verdict | +|---|---| +| Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for the consultation trigger | +| Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies | Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it | +| Ablation: CLAUDE.md-only arm holds sonnet 8/8 without the hook note | Confirms H2 sub-hypothesis (hook redundant); keep hook only if its removal regresses | +| Ablation: hook-only arm regresses vs CLAUDE.md-only | Channel strength ordering: persistent-context > session-start injection | +| Haiku moves off 0/8 with the winning sonnet packaging | H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover | +| Haiku stays 0/8 after ~3 iterations of the sonnet-winning wording | Capability floor for unprompted triggering at haiku tier — record it and stop iterating | +| W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains | +| Any target cell moves on <2 of 3 reps | Noise, not signal — do not accept the change (variance demonstrated 2026-07-03/04) | + +## Meta-lesson (why this note exists) + +Track hypothesis → expected-outcome mapping **before** running a wording loop. It forces +each grid cell to have a declared meaning, prevents post-hoc rationalization of noisy +flips, and lets later evals zero in faster by reusing confirmed mechanisms (e.g., if H1 +confirms here, start the *next* plugin's unprompted-behavior wording at trigger-conditioned +phrasing instead of rediscovering it). Pair this note's pattern with the procedure in +[[running-autoresearch-skill-evals]]. + +## Status + +- 2026-07-04: hypotheses locked, experiment starting (baseline commit cc-os `c645faa`). + Results to be appended here per iteration. diff --git a/reference/bash-pipefail-silent-exit-gotcha.md b/reference/bash-pipefail-silent-exit-gotcha.md new file mode 100644 index 0000000..7e7187d --- /dev/null +++ b/reference/bash-pipefail-silent-exit-gotcha.md @@ -0,0 +1,59 @@ +--- +type: reference +title: Bash pipefail silent-exit gotcha (grep in command substitution) +summary: Under set -euo pipefail, VAR=$(grep ... | ...) silently kills the script when grep finds no match — and the debugging lesson that "works interactively, fails in cron" claims must be tested by re-running the full script under its own shell options. +tags: + - scope/global + - type/reference + - domain/bash + - domain/debugging +--- + +# Bash pipefail silent-exit gotcha + +## The failure mode + +Under `set -euo pipefail`, a variable assignment via command substitution that +contains a grep pipeline: + +```bash +VAR=$(grep -oE 'pattern' "$FILE" 2>/dev/null | tail -1 | awk '{print $2}') +``` + +**silently terminates the script when grep finds no match:** + +1. No match → `grep` exits 1 +2. `pipefail` propagates that through the pipeline (`tail`/`awk` exiting 0 doesn't rescue it) +3. The assignment's exit status is the substitution's exit status → nonzero +4. `set -e` exits the script — with **zero error output** (`set -e` prints nothing, + and here grep's stderr was `/dev/null`'d anyway) + +**Symptom signature:** script halts silently mid-run, no error anywhere in any log, +everything before the halt point succeeded. + +**Guard pattern:** + +```bash +VAR=$(pipeline || true) # || true inside the substitution rescues pipefail +``` + +Or delete the line if it's dead code (a value that can never match on that host). + +## The debugging lesson (bigger than the gotcha) + +Confirmed in production 2026-07-04: the ovh-prod `backup.sh` halted silently every +night for ~2 months (2026-05-08 → 2026-07-04) and was **misattributed to a +cron-vs-interactive environment difference** across multiple debugging sessions. +Why the misattribution survived: each suspect command (`find -delete`, the notifier) +was re-run individually over SSH and worked — but the grep pipeline was never +re-tested **under the script's own shell options**. The failure had nothing to do +with cron; it reproduced identically in an interactive shell the moment the exact +pipeline was run under `set -euo pipefail`. + +**Rule:** when a script "works interactively but fails under cron/systemd", before +blaming the environment, re-run the *full script* (or the exact failing region under +the same `set -euo pipefail` flags) interactively. Testing commands in isolation +strips the shell-option context that may be the actual killer. Cheap first check: +`bash -c 'set -euo pipefail; ; echo REACHED'`. + +Incident write-up: `~/servers/ovh-prod/docs/incidents.md`, 2026-07-04 entry.