Netflix · Hollow 2026
Incremental dataset validation
Optimizing our duplicate key check to operate on only the changed records, while preserving correctness.
- My role
- Design, implementation, correctness testing, and rollout
- Outcome
- ~28–40s → ~1.5s validation on a critical production dataset
The dataset
Check For Duplicates Among What Changed
Within the changes
New keys compared with each other.
Against existing records
Look up keys in the previous index.
The problem
Hollow's duplicate-primary-key validator protects consumers from invalid datasets. On a dataset in the content delivery critical path, this check consumed about 28–40 seconds of a roughly three-minute cycle.
The design
I built an incremental validator around a primary-key index representing the last committed cycle. Comparing newly populated ordinals with previous ordinals lets the validator concentrate on changes.
The important constraint is keeping the index accurate across cycles. An incremental check is only correct when the state it reuses is correct; stale or corrupted index state would invalidate the optimization.
Validation and rollout
The implementation used an older code path that needed careful examination. I benchmarked it and tested for race conditions and corruption, integrated it with internal teams, and exercised it on a live follow namespace mirroring the target dataset.
The result
Validation on the target dataset dropped to about 1.5 seconds per cycle. This addressed a separate bottleneck alongside the Gutenberg propagation work.
Background & references
Public documentation for the systems discussed here.