← Overview

Netflix · Gutenberg 2026

Fast dataset propagation

Getting new dataset versions to consumers sooner, without giving up the controls that make versioned data safe to operate.

My role
Design, implementation, benchmarking, and rollout
Outcome
~30s → ~1s measured publish-to-consumption latency
Version propagation

Before Polling only

PersistRefresh~7 secondsWrite ↑Poll 30 secondsPublisher serviceGutenbergpublisher stackVersion writesCassandraGutenbergconsumer stackCachedtopic stateClient application

~30–37 seconds to receive an update.

After Polling + notifications

PersistRefresh~7 secondsWrite ↑Poll 1 secondPublisher serviceGutenbergpublisher stackVWatch leadersL1L2L3Topic replicationCassandraPer-topiclong poll ←Notify →Best effortGutenbergconsumer stackCache + updatedtopic stateClient application

~1s to receive an update.

The durable write and periodic cache refresh remain. VWatch adds a best-effort notification path after persistence. Animation is illustrative; labels show production intervals.

The problem

Gutenberg distributes versioned datasets between services. A server-side cache and periodic client polling introduced delay between publishing a version and consuming it. When a workflow chained multiple producers and consumers, those delays accumulated.

The content delivery team needed faster propagation for its Fast Turnaround Availability work. I owned the Gutenberg changes that addressed this part of the pipeline.

The design

I integrated VWatch into Gutenberg's publisher and consumer paths. After persisting a version in Cassandra, a publisher sends a best-effort notification. Consumer-side topic trackers listen for updates and make the newer state available to reads.

The last-sync time acts as a fencing token. Notifications improve freshness, while the persisted version remains the durable foundation. The design also has to preserve existing semantics such as pinning a topic to an older version.

Getting it into production

I benchmarked CPU, memory, and load, and partnered with VWatch's developer to address the behavior of this new use case. Production scale exposed bottlenecks and race conditions that were not visible in early testing.

A rollout uncovered a pinning edge case. I expanded test coverage and added shadow validation to compare behavior before increasing exposure. Correct version selection mattered as much as the latency improvement.

The result

Measured end-to-end propagation fell from roughly 30 seconds to roughly one second. The change improved a shared platform path rather than requiring every client team to implement its own optimization.

Background & references

Public documentation for the systems discussed here.