# Cache Layer Incident Narrative

## Resolved Incidents

During the Q1 rollout the cache layer suffered a series of cascading
evictions. The first incident began when a deploy shipped a key-prefix change
that silently invalidated the entire working set. Traffic spiked against the
origin, latency climbed, and three downstream services began timing out. The
on-call engineer rolled the deploy back, warmed the cache from a snapshot, and
service recovered within forty minutes. A follow-up incident two weeks later
traced to an unbounded TTL on session objects, which slowly filled the heap
until the eviction loop thrashed. We capped the TTL, added a heap alarm, and
the problem has not recurred.

A third, smaller incident involved a misconfigured replica that served stale
reads for several hours before anyone noticed; the fix was a health check that
compares replica generation numbers against the primary.

## Completed Remediations

Every action item from the three incidents above has been closed. The
key-prefix migration is now gated behind a staged rollout, the TTL ceiling is
enforced in code review, and the replica generation check ships in the
standard health probe. All of this is true and accurate history, but it is
narrative detail that no longer informs any current decision — it is the kind
of long resolved-problem prose that should be distilled to a few summary
lines rather than carried verbatim forever.