166 lines
7.8 KiB
Markdown
166 lines
7.8 KiB
Markdown
# Incident log
|
||
|
||
Dated entries for diagnosed-and-fixed system incidents on this machine.
|
||
Newest first. Keep entries factual: symptom, root cause, what was ruled out,
|
||
fix applied, and any follow-up.
|
||
|
||
## 2026-06-26 — memtest86+ confirms bad RAM (root cause of the 2026-06-24 Btrfs corruption)
|
||
|
||
**Result: FAILED.** Ran memtest86+ v8.10 from the GRUB boot entry on the
|
||
Ryzen 5 5600G / 95.3 GB DDR4-2666 (CAS 20-19-19-43) workstation.
|
||
|
||
- **Status: Failed!, Errors: 7** at only **0:00:51** elapsed — still in Pass 0,
|
||
Test #3 (Moving inversions, 1s & 0s). Errored almost immediately.
|
||
- Both captured failures are **single-bit flips** (signature of a failing
|
||
module, not a controller-wide / addressing fault):
|
||
- `0x000bc0115dd0` (47 GB): expected `f…`, found `bf…` → **bit 62 flipped 1→0**
|
||
- `0x001005f391e0` (64 GB): expected `0…`, found `…01000000` → **bit 24 flipped 0→1**
|
||
- Temps 58/65 °C (fine). DDR4-2666 is JEDEC-standard, **not** an aggressive
|
||
XMP/EXPO overclock — so this is not merely an unstable memory profile.
|
||
|
||
**Conclusion**
|
||
|
||
Confirms the 2026-06-24 hypothesis: scattered multi-file Btrfs csum corruption
|
||
with zero drive I/O errors and clean NVMe SMART = **bad RAM** writing garbage to
|
||
otherwise-good flash. Root cause: **confirmed**.
|
||
|
||
**RAM layout (from `dmidecode -t memory`, 2026-06-26)**
|
||
|
||
96 GB total = **two complete G.Skill Ripjaws V kits**, both DDR4-3200 CL16
|
||
running at 2666 (XMP/DOCP off):
|
||
|
||
| Slot | Size | Part number | Kit |
|
||
|------|------|-------------|-----|
|
||
| Channel A · DIMM 0 | 32 GB | F4-3200C16-32GVK | 2×32 GB pair → 64 GB |
|
||
| Channel B · DIMM 0 | 32 GB | F4-3200C16-32GVK | (matched cross-channel) |
|
||
| Channel A · DIMM 1 | 16 GB | F4-3200C16-16GVK | 2×16 GB pair → 32 GB |
|
||
| Channel B · DIMM 1 | 16 GB | F4-3200C16-16GVK | (matched cross-channel) |
|
||
|
||
Two valid matched dual-channel pairs → whichever stick fails, drop to the other
|
||
complete pair and keep dual-channel. **Fallbacks need zero purchase:** if a
|
||
16 GB stick is bad → run the 2×32 = 64 GB; if a 32 GB stick is bad → run the
|
||
2×16 = 32 GB.
|
||
|
||
**Booting memtest (GRUB hotkeys unreliable on this box)**
|
||
|
||
memtest86+ 8.10 is installed (`/boot/memtest86+x64.efi`); GRUB uses
|
||
`GRUB_DEFAULT=saved`, so one-shot boot needs no keypresses:
|
||
|
||
```bash
|
||
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
|
||
sudo grep -o "menuentry '[^']*emtest[^']*'" /boot/grub2/grub.cfg # get exact title
|
||
sudo grub2-reboot "<exact title>" # one-shot; reverts after next boot
|
||
sudo systemctl poweroff # pull sticks, then power on → boots memtest
|
||
```
|
||
|
||
Exit memtest with **Esc** (reboots back to Fedora).
|
||
|
||
**Follow-up / next steps**
|
||
|
||
- **Treat the machine as untrustworthy until the RAM is replaced** — any data
|
||
written (including backups) since the corruption began may be silently bad.
|
||
Be cautious about pruning known-good restic snapshots.
|
||
- **Isolate the bad DIMM** — test **by kit first** (the 2×32, then the 2×16);
|
||
a passing pair is immediately a runnable config. Only split a *failing* pair
|
||
to find the single bad stick. Errors at 47 GB / 64 GB are interleaved logical
|
||
addresses and do NOT map to one physical stick.
|
||
- **Don't discard the bad stick** — G.Skill Ripjaws V carry a limited lifetime
|
||
warranty; RMA for a free replacement (restores the matched pair at no cost).
|
||
Keep its good partner as a spare or for the RMA-restored pair. Avoid buying a
|
||
single stick to pair (DDR4 matching is finicky) unless RMA fails.
|
||
- After fixing, **re-run memtest to a clean multi-pass**, then **re-scrub
|
||
`/home`** and watch for *new* csum errors. None appearing = corruption was
|
||
historical and is now resolved. Defer `restic check --read-data` until the RAM
|
||
is known-good (a bad-RAM box throws false integrity errors).
|
||
|
||
## 2026-06-24 — Chrome repeatedly crashing (SIGBUS), corrupt Favicons block on single-copy Btrfs
|
||
|
||
**Symptom**
|
||
|
||
Google Chrome crashed within seconds of launch, repeatedly. Three crashes in
|
||
three minutes on 2026-06-24 (13:15, 13:16, 13:17), all signal **SIGBUS**.
|
||
History of crashes back to April (mixed SIGTRAP / SIGILL / SIGSEGV / SIGBUS),
|
||
18 minidumps total.
|
||
|
||
**Root cause**
|
||
|
||
A single corrupt data block at offset **942080** of
|
||
`~/.config/google-chrome/Profile 2/Favicons` (a SQLite DB, inode 141150,
|
||
btrfs root 257). Chrome `mmap()`s the file on startup; the kernel raises SIGBUS
|
||
when it cannot fault in the bad page.
|
||
|
||
Each SIGBUS coredump correlated to the second with a kernel log line:
|
||
|
||
```
|
||
BTRFS warning: csum failed root 257 ino 141150 off 942080 csum 0x0858109c expected 0x0858109e mirror 1
|
||
```
|
||
|
||
Same single-bit flip every time (`9c` vs `9e`). The volume is `Data, single`
|
||
(no mirror), so Btrfs detects but **cannot self-heal**. `corruption_errs` was
|
||
climbing (41 → 42). Disk `read` / `write` / `flush` `io_errs` were all 0 —
|
||
silent at-rest corruption; the NVMe is not reporting hardware faults.
|
||
|
||
**Ruled out**
|
||
|
||
- **OOM** — 87 GiB RAM free, and SIGBUS ≠ SIGKILL.
|
||
- **Disk full** — 5% used.
|
||
- **Chrome version / update** — install dated 2026-06-20 predates the crashes;
|
||
`rpm -V` clean.
|
||
- **Profile / extension corruption.**
|
||
- **Dual-GPU / Wayland stack** — GPU libs in the dump were merely mapped; the
|
||
`gpu_channel.cc Buffer Handle is null` line was a downstream symptom logged
|
||
~2s after the SIGBUS.
|
||
|
||
**Fix applied (2026-06-24)**
|
||
|
||
Confirmed Chrome not running, then deleted the regenerable cache files
|
||
`Favicons` and `Favicons-journal` from `Profile 2/`. Chrome regenerates these on
|
||
next launch. Verified gone.
|
||
|
||
**Scrub result (2026-06-24 13:28)**
|
||
|
||
`sudo btrfs scrub start -B /home` finished in 0:55, scanned 79.57 GiB.
|
||
**Error summary: `csum=2`, Corrected 0, Uncorrectable 2.** The corruption is
|
||
**not** isolated to the Favicons file — two further uncorrectable blocks, in
|
||
two different subvolumes, both regenerable/reinstallable:
|
||
|
||
| File | Subvol | Action |
|
||
|------|--------|--------|
|
||
| `~/.cache/google-chrome/Profile 1/Cache/Cache_Data/5f188fe6012a3cdc_0` | home (root 257) | Deleted (regenerates) ✅ |
|
||
| `/usr/share/help/fr/gdm/index.docbook` | root (root 259) | Owned by `gdm-49.2-2.fc43`; restore via `sudo dnf reinstall gdm` |
|
||
|
||
`corruption_errs` 44; `read`/`write`/`flush` `io_errs` 0.
|
||
|
||
**Remediation + verification (2026-06-24 13:38–13:42)**
|
||
|
||
- Deleted the Chrome cache block (`rm`, regenerates).
|
||
- Restored the gdm file: `sudo dnf reinstall gdm`.
|
||
- **SSD health is clean** — `sudo nvme smart-log /dev/nvme0n1`:
|
||
`critical_warning 0`, `media_errors 0`, `percentage_used 0%`. The drive has
|
||
logged **zero** media errors at the hardware level → a *failing SSD* is
|
||
unlikely; corruption reached good flash from upstream (RAM/bus).
|
||
- **Second scrub (13:41)** still reports `csum=2` uncorrectable — but at the
|
||
**exact same two physical blocks** as the first scrub (`logical 1754923008`,
|
||
`logical 21439315968`), and this time **with no resolvable path**. The live
|
||
files now point at fresh extents; the corrupt extents are **orphaned dead
|
||
blocks** (not snapshot-pinned — `snapper`/`timeshift` not installed).
|
||
|
||
**Assessment**
|
||
|
||
Same two static blocks across two scrubs 13 min apart = corruption is **static,
|
||
not actively spreading** (no new blocks appeared). Combined with the clean SSD
|
||
SMART, this most likely stems from a **single past corruption event** rather
|
||
than an actively-failing component. The crash-causing live files are fixed;
|
||
the residual scrub errors affect no live file.
|
||
|
||
**Outstanding / follow-up**
|
||
|
||
- **Re-scrub in a day or two** (`sudo btrfs scrub start -B /home`). If the two
|
||
orphaned blocks have cleared (reclaimed) and no *new* errors appear → fully
|
||
resolved. **New** blocks appearing = corruption is ongoing → back up now and
|
||
run memtest.
|
||
- **Run memtest86+** (Fedora GRUB boot entry) for confidence — scattered
|
||
multi-file corruption with zero drive I/O errors is the classic bad-RAM
|
||
signature, even if currently static.
|
||
- The `corruption_errs` counter will not drop on a single-copy volume — expected.
|