server-desktop-01/docs/incidents.md

97 lines
4.3 KiB
Markdown
Raw Normal View History

# Incident log
Dated entries for diagnosed-and-fixed system incidents on this machine.
Newest first. Keep entries factual: symptom, root cause, what was ruled out,
fix applied, and any follow-up.
## 2026-06-24 — Chrome repeatedly crashing (SIGBUS), corrupt Favicons block on single-copy Btrfs
**Symptom**
Google Chrome crashed within seconds of launch, repeatedly. Three crashes in
three minutes on 2026-06-24 (13:15, 13:16, 13:17), all signal **SIGBUS**.
History of crashes back to April (mixed SIGTRAP / SIGILL / SIGSEGV / SIGBUS),
18 minidumps total.
**Root cause**
A single corrupt data block at offset **942080** of
`~/.config/google-chrome/Profile 2/Favicons` (a SQLite DB, inode 141150,
btrfs root 257). Chrome `mmap()`s the file on startup; the kernel raises SIGBUS
when it cannot fault in the bad page.
Each SIGBUS coredump correlated to the second with a kernel log line:
```
BTRFS warning: csum failed root 257 ino 141150 off 942080 csum 0x0858109c expected 0x0858109e mirror 1
```
Same single-bit flip every time (`9c` vs `9e`). The volume is `Data, single`
(no mirror), so Btrfs detects but **cannot self-heal**. `corruption_errs` was
climbing (41 → 42). Disk `read` / `write` / `flush` `io_errs` were all 0 —
silent at-rest corruption; the NVMe is not reporting hardware faults.
**Ruled out**
- **OOM** — 87 GiB RAM free, and SIGBUS ≠ SIGKILL.
- **Disk full** — 5% used.
- **Chrome version / update** — install dated 2026-06-20 predates the crashes;
`rpm -V` clean.
- **Profile / extension corruption.**
- **Dual-GPU / Wayland stack** — GPU libs in the dump were merely mapped; the
`gpu_channel.cc Buffer Handle is null` line was a downstream symptom logged
~2s after the SIGBUS.
**Fix applied (2026-06-24)**
Confirmed Chrome not running, then deleted the regenerable cache files
`Favicons` and `Favicons-journal` from `Profile 2/`. Chrome regenerates these on
next launch. Verified gone.
**Scrub result (2026-06-24 13:28)**
`sudo btrfs scrub start -B /home` finished in 0:55, scanned 79.57 GiB.
**Error summary: `csum=2`, Corrected 0, Uncorrectable 2.** The corruption is
**not** isolated to the Favicons file — two further uncorrectable blocks, in
two different subvolumes, both regenerable/reinstallable:
| File | Subvol | Action |
|------|--------|--------|
| `~/.cache/google-chrome/Profile 1/Cache/Cache_Data/5f188fe6012a3cdc_0` | home (root 257) | Deleted (regenerates) ✅ |
| `/usr/share/help/fr/gdm/index.docbook` | root (root 259) | Owned by `gdm-49.2-2.fc43`; restore via `sudo dnf reinstall gdm` |
`corruption_errs` 44; `read`/`write`/`flush` `io_errs` 0.
**Remediation + verification (2026-06-24 13:3813:42)**
- Deleted the Chrome cache block (`rm`, regenerates).
- Restored the gdm file: `sudo dnf reinstall gdm`.
- **SSD health is clean** — `sudo nvme smart-log /dev/nvme0n1`:
`critical_warning 0`, `media_errors 0`, `percentage_used 0%`. The drive has
logged **zero** media errors at the hardware level → a *failing SSD* is
unlikely; corruption reached good flash from upstream (RAM/bus).
- **Second scrub (13:41)** still reports `csum=2` uncorrectable — but at the
**exact same two physical blocks** as the first scrub (`logical 1754923008`,
`logical 21439315968`), and this time **with no resolvable path**. The live
files now point at fresh extents; the corrupt extents are **orphaned dead
blocks** (not snapshot-pinned — `snapper`/`timeshift` not installed).
**Assessment**
Same two static blocks across two scrubs 13 min apart = corruption is **static,
not actively spreading** (no new blocks appeared). Combined with the clean SSD
SMART, this most likely stems from a **single past corruption event** rather
than an actively-failing component. The crash-causing live files are fixed;
the residual scrub errors affect no live file.
**Outstanding / follow-up**
- **Re-scrub in a day or two** (`sudo btrfs scrub start -B /home`). If the two
orphaned blocks have cleared (reclaimed) and no *new* errors appear → fully
resolved. **New** blocks appearing = corruption is ongoing → back up now and
run memtest.
- **Run memtest86+** (Fedora GRUB boot entry) for confidence — scattered
multi-file corruption with zero drive I/O errors is the classic bad-RAM
signature, even if currently static.
- The `corruption_errs` counter will not drop on a single-copy volume — expected.