server-desktop-01/docs/incidents.md

97 lines
4.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Incident log
Dated entries for diagnosed-and-fixed system incidents on this machine.
Newest first. Keep entries factual: symptom, root cause, what was ruled out,
fix applied, and any follow-up.
## 2026-06-24 — Chrome repeatedly crashing (SIGBUS), corrupt Favicons block on single-copy Btrfs
**Symptom**
Google Chrome crashed within seconds of launch, repeatedly. Three crashes in
three minutes on 2026-06-24 (13:15, 13:16, 13:17), all signal **SIGBUS**.
History of crashes back to April (mixed SIGTRAP / SIGILL / SIGSEGV / SIGBUS),
18 minidumps total.
**Root cause**
A single corrupt data block at offset **942080** of
`~/.config/google-chrome/Profile 2/Favicons` (a SQLite DB, inode 141150,
btrfs root 257). Chrome `mmap()`s the file on startup; the kernel raises SIGBUS
when it cannot fault in the bad page.
Each SIGBUS coredump correlated to the second with a kernel log line:
```
BTRFS warning: csum failed root 257 ino 141150 off 942080 csum 0x0858109c expected 0x0858109e mirror 1
```
Same single-bit flip every time (`9c` vs `9e`). The volume is `Data, single`
(no mirror), so Btrfs detects but **cannot self-heal**. `corruption_errs` was
climbing (41 → 42). Disk `read` / `write` / `flush` `io_errs` were all 0 —
silent at-rest corruption; the NVMe is not reporting hardware faults.
**Ruled out**
- **OOM** — 87 GiB RAM free, and SIGBUS ≠ SIGKILL.
- **Disk full** — 5% used.
- **Chrome version / update** — install dated 2026-06-20 predates the crashes;
`rpm -V` clean.
- **Profile / extension corruption.**
- **Dual-GPU / Wayland stack** — GPU libs in the dump were merely mapped; the
`gpu_channel.cc Buffer Handle is null` line was a downstream symptom logged
~2s after the SIGBUS.
**Fix applied (2026-06-24)**
Confirmed Chrome not running, then deleted the regenerable cache files
`Favicons` and `Favicons-journal` from `Profile 2/`. Chrome regenerates these on
next launch. Verified gone.
**Scrub result (2026-06-24 13:28)**
`sudo btrfs scrub start -B /home` finished in 0:55, scanned 79.57 GiB.
**Error summary: `csum=2`, Corrected 0, Uncorrectable 2.** The corruption is
**not** isolated to the Favicons file — two further uncorrectable blocks, in
two different subvolumes, both regenerable/reinstallable:
| File | Subvol | Action |
|------|--------|--------|
| `~/.cache/google-chrome/Profile 1/Cache/Cache_Data/5f188fe6012a3cdc_0` | home (root 257) | Deleted (regenerates) ✅ |
| `/usr/share/help/fr/gdm/index.docbook` | root (root 259) | Owned by `gdm-49.2-2.fc43`; restore via `sudo dnf reinstall gdm` |
`corruption_errs` 44; `read`/`write`/`flush` `io_errs` 0.
**Remediation + verification (2026-06-24 13:3813:42)**
- Deleted the Chrome cache block (`rm`, regenerates).
- Restored the gdm file: `sudo dnf reinstall gdm`.
- **SSD health is clean** — `sudo nvme smart-log /dev/nvme0n1`:
`critical_warning 0`, `media_errors 0`, `percentage_used 0%`. The drive has
logged **zero** media errors at the hardware level → a *failing SSD* is
unlikely; corruption reached good flash from upstream (RAM/bus).
- **Second scrub (13:41)** still reports `csum=2` uncorrectable — but at the
**exact same two physical blocks** as the first scrub (`logical 1754923008`,
`logical 21439315968`), and this time **with no resolvable path**. The live
files now point at fresh extents; the corrupt extents are **orphaned dead
blocks** (not snapshot-pinned — `snapper`/`timeshift` not installed).
**Assessment**
Same two static blocks across two scrubs 13 min apart = corruption is **static,
not actively spreading** (no new blocks appeared). Combined with the clean SSD
SMART, this most likely stems from a **single past corruption event** rather
than an actively-failing component. The crash-causing live files are fixed;
the residual scrub errors affect no live file.
**Outstanding / follow-up**
- **Re-scrub in a day or two** (`sudo btrfs scrub start -B /home`). If the two
orphaned blocks have cleared (reclaimed) and no *new* errors appear → fully
resolved. **New** blocks appearing = corruption is ongoing → back up now and
run memtest.
- **Run memtest86+** (Fedora GRUB boot entry) for confidence — scattered
multi-file corruption with zero drive I/O errors is the classic bad-RAM
signature, even if currently static.
- The `corruption_errs` counter will not drop on a single-copy volume — expected.