28 lines
1.4 KiB
Markdown
28 lines
1.4 KiB
Markdown
|
# Cache Layer Incident Narrative
|
||
|
|
|
||
|
|
## Resolved Incidents
|
||
|
|
|
||
|
|
During the Q1 rollout the cache layer suffered a series of cascading
|
||
|
|
evictions. The first incident began when a deploy shipped a key-prefix change
|
||
|
|
that silently invalidated the entire working set. Traffic spiked against the
|
||
|
|
origin, latency climbed, and three downstream services began timing out. The
|
||
|
|
on-call engineer rolled the deploy back, warmed the cache from a snapshot, and
|
||
|
|
service recovered within forty minutes. A follow-up incident two weeks later
|
||
|
|
traced to an unbounded TTL on session objects, which slowly filled the heap
|
||
|
|
until the eviction loop thrashed. We capped the TTL, added a heap alarm, and
|
||
|
|
the problem has not recurred.
|
||
|
|
|
||
|
|
A third, smaller incident involved a misconfigured replica that served stale
|
||
|
|
reads for several hours before anyone noticed; the fix was a health check that
|
||
|
|
compares replica generation numbers against the primary.
|
||
|
|
|
||
|
|
## Completed Remediations
|
||
|
|
|
||
|
|
Every action item from the three incidents above has been closed. The
|
||
|
|
key-prefix migration is now gated behind a staged rollout, the TTL ceiling is
|
||
|
|
enforced in code review, and the replica generation check ships in the
|
||
|
|
standard health probe. All of this is true and accurate history, but it is
|
||
|
|
narrative detail that no longer informs any current decision — it is the kind
|
||
|
|
of long resolved-problem prose that should be distilled to a few summary
|
||
|
|
lines rather than carried verbatim forever.
|