60 lines
2.5 KiB
Markdown
60 lines
2.5 KiB
Markdown
---
|
|
type: reference
|
|
title: Bash pipefail silent-exit gotcha (grep in command substitution)
|
|
summary: Under set -euo pipefail, VAR=$(grep ... | ...) silently kills the script when grep finds no match — and the debugging lesson that "works interactively, fails in cron" claims must be tested by re-running the full script under its own shell options.
|
|
tags:
|
|
- scope/global
|
|
- type/reference
|
|
- domain/bash
|
|
- domain/debugging
|
|
---
|
|
|
|
# Bash pipefail silent-exit gotcha
|
|
|
|
## The failure mode
|
|
|
|
Under `set -euo pipefail`, a variable assignment via command substitution that
|
|
contains a grep pipeline:
|
|
|
|
```bash
|
|
VAR=$(grep -oE 'pattern' "$FILE" 2>/dev/null | tail -1 | awk '{print $2}')
|
|
```
|
|
|
|
**silently terminates the script when grep finds no match:**
|
|
|
|
1. No match → `grep` exits 1
|
|
2. `pipefail` propagates that through the pipeline (`tail`/`awk` exiting 0 doesn't rescue it)
|
|
3. The assignment's exit status is the substitution's exit status → nonzero
|
|
4. `set -e` exits the script — with **zero error output** (`set -e` prints nothing,
|
|
and here grep's stderr was `/dev/null`'d anyway)
|
|
|
|
**Symptom signature:** script halts silently mid-run, no error anywhere in any log,
|
|
everything before the halt point succeeded.
|
|
|
|
**Guard pattern:**
|
|
|
|
```bash
|
|
VAR=$(pipeline || true) # || true inside the substitution rescues pipefail
|
|
```
|
|
|
|
Or delete the line if it's dead code (a value that can never match on that host).
|
|
|
|
## The debugging lesson (bigger than the gotcha)
|
|
|
|
Confirmed in production 2026-07-04: the ovh-prod `backup.sh` halted silently every
|
|
night for ~2 months (2026-05-08 → 2026-07-04) and was **misattributed to a
|
|
cron-vs-interactive environment difference** across multiple debugging sessions.
|
|
Why the misattribution survived: each suspect command (`find -delete`, the notifier)
|
|
was re-run individually over SSH and worked — but the grep pipeline was never
|
|
re-tested **under the script's own shell options**. The failure had nothing to do
|
|
with cron; it reproduced identically in an interactive shell the moment the exact
|
|
pipeline was run under `set -euo pipefail`.
|
|
|
|
**Rule:** when a script "works interactively but fails under cron/systemd", before
|
|
blaming the environment, re-run the *full script* (or the exact failing region under
|
|
the same `set -euo pipefail` flags) interactively. Testing commands in isolation
|
|
strips the shell-option context that may be the actual killer. Cheap first check:
|
|
`bash -c 'set -euo pipefail; <suspect line>; echo REACHED'`.
|
|
|
|
Incident write-up: `~/servers/ovh-prod/docs/incidents.md`, 2026-07-04 entry.
|