server-desktop-01/docs/incidents.md

Incident log

Dated entries for diagnosed-and-fixed system incidents on this machine. Newest first. Keep entries factual: symptom, root cause, what was ruled out, fix applied, and any follow-up.

2026-06-26 — memtest86+ confirms bad RAM (root cause of the 2026-06-24 Btrfs corruption)

Result: FAILED. Ran memtest86+ v8.10 from the GRUB boot entry on the Ryzen 5 5600G / 95.3 GB DDR4-2666 (CAS 20-19-19-43) workstation.

• Status: Failed!, Errors: 7 at only 0:00:51 elapsed — still in Pass 0, Test #3 (Moving inversions, 1s & 0s). Errored almost immediately.
• Both captured failures are single-bit flips (signature of a failing module, not a controller-wide / addressing fault):
  • 0x000bc0115dd0 (47 GB): expected f…, found bf… → bit 62 flipped 1→0
  • 0x001005f391e0 (64 GB): expected 0…, found …01000000 → bit 24 flipped 0→1
• Temps 58/65 °C (fine). DDR4-2666 is JEDEC-standard, not an aggressive XMP/EXPO overclock — so this is not merely an unstable memory profile.

Conclusion

Confirms the 2026-06-24 hypothesis: scattered multi-file Btrfs csum corruption with zero drive I/O errors and clean NVMe SMART = bad RAM writing garbage to otherwise-good flash. Root cause: confirmed.

RAM layout (from dmidecode -t memory, 2026-06-26)

96 GB total = two complete G.Skill Ripjaws V kits, both DDR4-3200 CL16 running at 2666 (XMP/DOCP off):

Slot	Size	Part number	Kit
Channel A · DIMM 0	32 GB	F4-3200C16-32GVK	2×32 GB pair → 64 GB
Channel B · DIMM 0	32 GB	F4-3200C16-32GVK	(matched cross-channel)
Channel A · DIMM 1	16 GB	F4-3200C16-16GVK	2×16 GB pair → 32 GB
Channel B · DIMM 1	16 GB	F4-3200C16-16GVK	(matched cross-channel)

Two valid matched dual-channel pairs → whichever stick fails, drop to the other complete pair and keep dual-channel. Fallbacks need zero purchase: if a 16 GB stick is bad → run the 2×32 = 64 GB; if a 32 GB stick is bad → run the 2×16 = 32 GB.

Booting memtest (GRUB hotkeys unreliable on this box)

memtest86+ 8.10 is installed (/boot/memtest86+x64.efi); GRUB uses GRUB_DEFAULT=saved, so one-shot boot needs no keypresses:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grep -o "menuentry '[^']*emtest[^']*'" /boot/grub2/grub.cfg   # get exact title
sudo grub2-reboot "<exact title>"   # one-shot; reverts after next boot
sudo systemctl poweroff             # pull sticks, then power on → boots memtest

Exit memtest with Esc (reboots back to Fedora).

Follow-up / next steps

• Treat the machine as untrustworthy until the RAM is replaced — any data written (including backups) since the corruption began may be silently bad. Be cautious about pruning known-good restic snapshots.
• Isolate the bad DIMM — test by kit first (the 2×32, then the 2×16); a passing pair is immediately a runnable config. Only split a failing pair to find the single bad stick. Errors at 47 GB / 64 GB are interleaved logical addresses and do NOT map to one physical stick.
• Don't discard the bad stick — G.Skill Ripjaws V carry a limited lifetime warranty; RMA for a free replacement (restores the matched pair at no cost). Keep its good partner as a spare or for the RMA-restored pair. Avoid buying a single stick to pair (DDR4 matching is finicky) unless RMA fails.
• After fixing, re-run memtest to a clean multi-pass, then re-scrub /home and watch for new csum errors. None appearing = corruption was historical and is now resolved. Defer restic check --read-data until the RAM is known-good (a bad-RAM box throws false integrity errors).

2026-06-24 — Chrome repeatedly crashing (SIGBUS), corrupt Favicons block on single-copy Btrfs

Symptom

Google Chrome crashed within seconds of launch, repeatedly. Three crashes in three minutes on 2026-06-24 (13:15, 13:16, 13:17), all signal SIGBUS. History of crashes back to April (mixed SIGTRAP / SIGILL / SIGSEGV / SIGBUS), 18 minidumps total.

Root cause

A single corrupt data block at offset 942080 of ~/.config/google-chrome/Profile 2/Favicons (a SQLite DB, inode 141150, btrfs root 257). Chrome mmap()s the file on startup; the kernel raises SIGBUS when it cannot fault in the bad page.

Each SIGBUS coredump correlated to the second with a kernel log line:

BTRFS warning: csum failed root 257 ino 141150 off 942080 csum 0x0858109c expected 0x0858109e mirror 1

Same single-bit flip every time (9c vs 9e). The volume is Data, single (no mirror), so Btrfs detects but cannot self-heal. corruption_errs was climbing (41 → 42). Disk read / write / flush io_errs were all 0 — silent at-rest corruption; the NVMe is not reporting hardware faults.

Ruled out

• OOM — 87 GiB RAM free, and SIGBUS ≠ SIGKILL.
• Disk full — 5% used.
• Chrome version / update — install dated 2026-06-20 predates the crashes; rpm -V clean.
• Profile / extension corruption.
• Dual-GPU / Wayland stack — GPU libs in the dump were merely mapped; the gpu_channel.cc Buffer Handle is null line was a downstream symptom logged ~2s after the SIGBUS.

Fix applied (2026-06-24)

Confirmed Chrome not running, then deleted the regenerable cache files Favicons and Favicons-journal from Profile 2/. Chrome regenerates these on next launch. Verified gone.

Scrub result (2026-06-24 13:28)

sudo btrfs scrub start -B /home finished in 0:55, scanned 79.57 GiB. Error summary: csum=2, Corrected 0, Uncorrectable 2. The corruption is not isolated to the Favicons file — two further uncorrectable blocks, in two different subvolumes, both regenerable/reinstallable:

File	Subvol	Action
~/.cache/google-chrome/Profile 1/Cache/Cache_Data/5f188fe6012a3cdc_0	home (root 257)	Deleted (regenerates) ✅
/usr/share/help/fr/gdm/index.docbook	root (root 259)	Owned by gdm-49.2-2.fc43; restore via sudo dnf reinstall gdm

corruption_errs 44; read/write/flush io_errs 0.

Remediation + verification (2026-06-24 13:38–13:42)

• Deleted the Chrome cache block (rm, regenerates).
• Restored the gdm file: sudo dnf reinstall gdm.
• SSD health is clean — sudo nvme smart-log /dev/nvme0n1: critical_warning 0, media_errors 0, percentage_used 0%. The drive has logged zero media errors at the hardware level → a failing SSD is unlikely; corruption reached good flash from upstream (RAM/bus).
• Second scrub (13:41) still reports csum=2 uncorrectable — but at the exact same two physical blocks as the first scrub (logical 1754923008, logical 21439315968), and this time with no resolvable path. The live files now point at fresh extents; the corrupt extents are orphaned dead blocks (not snapshot-pinned — snapper/timeshift not installed).

Assessment

Same two static blocks across two scrubs 13 min apart = corruption is static, not actively spreading (no new blocks appeared). Combined with the clean SSD SMART, this most likely stems from a single past corruption event rather than an actively-failing component. The crash-causing live files are fixed; the residual scrub errors affect no live file.

Outstanding / follow-up

• Re-scrub in a day or two (sudo btrfs scrub start -B /home). If the two orphaned blocks have cleared (reclaimed) and no new errors appear → fully resolved. New blocks appearing = corruption is ongoing → back up now and run memtest.
• Run memtest86+ (Fedora GRUB boot entry) for confidence — scattered multi-file corruption with zero drive I/O errors is the classic bad-RAM signature, even if currently static.
• The corruption_errs counter will not drop on a single-copy volume — expected.