Netflix · EVCache / Memcached 2026
Operating distributed caches
Keeping a replicated cache fleet healthy, and making its on-call rotation more sustainable.
- My role
- Fleet operations, on-call, defect fixes, and alerting improvements
- Outcome
- ~100 clusters · typically four regions · ~94% fewer after-hours pages
- Application clients
- Zone-local cache nodes
- Replicas across zones
The operating environment
The team operates roughly 100 EVCache clusters, typically replicated across four AWS regions. My work includes deployments, scaling, right-sizing, cache warming, and maintenance of Memcached-based systems.
I joined this rotation after team attrition, while continuing to support the in-memory dataset services. These were two distinct operational domains: cluster and capacity work on one side, publishing and dataset pipelines on the other.
Making the signals useful
I catalogued pages, alerts, and escalations to distinguish real failures from unhelpful noise. Some alerts needed tuning; others were evidence of recurring defects that needed a code fix.
I shipped fixes across the EVCache client, write-ahead log, and proxy, and deployed newer Memcache versions to address known bugs. Documenting routine work and helping newer engineers ramp up also reduced dependence on a few specialists.
The result
Across the team's operational improvements, monthly after-hours pages fell from 487 in January 2026 to 27 in June. I contributed through defect fixes and better paging rigor; this was a shared outcome, not a measure of availability.
I also defined and instrumented escalation toil for an organization-wide availability review, making work that did not appear in page counts more visible.
Background & references
Public documentation for the systems discussed here.