Netflix · Key-Value Platform 2024 internship
Safer Cassandra cleanup
Finding oversized rows and cleaning them up without overwhelming a production datastore.
- My role
- Scanner design and implementation
- Outcome
- CPU-aware scanning and deletion spread over time
- Scan with backpressure
- Identify wide rows
- Assign spread TTLs
The problem
Rows with very large numbers of columns can degrade Cassandra performance. Finding them requires substantial scanning, which creates its own risk to production traffic.
Controlling the work
I built a background scanner with CPU-aware backpressure. It throttles scanning under cluster load so remediation work does not compete unchecked with live requests.
Deletion needed its own control. Removing a large number of records at once creates tombstones that can make reads expensive until compaction handles them. I implemented spread-TTL deletes: records are marked for deletion and assigned randomized expiration windows, while the key-value layer filters marked data from client results.
The tradeoff
Cleanup progresses over time instead of completing as one large burst. The design balances remediation throughput with the health of the system serving customer traffic.