SecondBrain/reference/bash-pipefail-silent-exit-gotcha.md

type	title	summary	tags
reference	Bash pipefail silent-exit gotcha (grep in command substitution)	Under set -euo pipefail, VAR=$(grep ... | ...) silently kills the script when grep finds no match — and the debugging lesson that "works interactively, fails in cron" claims must be tested by re-running the full script under its own shell options.	scope/global type/reference domain/bash domain/debugging

Bash pipefail silent-exit gotcha

The failure mode

Under set -euo pipefail, a variable assignment via command substitution that contains a grep pipeline:

VAR=$(grep -oE 'pattern' "$FILE" 2>/dev/null | tail -1 | awk '{print $2}')

silently terminates the script when grep finds no match:

1. No match → grep exits 1
2. pipefail propagates that through the pipeline (tail/awk exiting 0 doesn't rescue it)
3. The assignment's exit status is the substitution's exit status → nonzero
4. set -e exits the script — with zero error output (set -e prints nothing, and here grep's stderr was /dev/null'd anyway)

Symptom signature: script halts silently mid-run, no error anywhere in any log, everything before the halt point succeeded.

Guard pattern:

VAR=$(pipeline || true)   # || true inside the substitution rescues pipefail

Or delete the line if it's dead code (a value that can never match on that host).

The debugging lesson (bigger than the gotcha)

Confirmed in production 2026-07-04: the ovh-prod backup.sh halted silently every night for ~2 months (2026-05-08 → 2026-07-04) and was misattributed to a cron-vs-interactive environment difference across multiple debugging sessions. Why the misattribution survived: each suspect command (find -delete, the notifier) was re-run individually over SSH and worked — but the grep pipeline was never re-tested under the script's own shell options. The failure had nothing to do with cron; it reproduced identically in an interactive shell the moment the exact pipeline was run under set -euo pipefail.

Rule: when a script "works interactively but fails under cron/systemd", before blaming the environment, re-run the full script (or the exact failing region under the same set -euo pipefail flags) interactively. Testing commands in isolation strips the shell-option context that may be the actual killer. Cheap first check: bash -c 'set -euo pipefail; <suspect line>; echo REACHED'.

Incident write-up: ~/servers/ovh-prod/docs/incidents.md, 2026-07-04 entry.