← Overview

Netflix · Hollow ecosystem 2025–2026

Recoverable datastore deletion

Making a datastore fully deletable—and making a mistaken deletion recoverable.

My role
End-to-end design and implementation
Outcome
~$270K annual infrastructure savings · two production recoveries
  1. Verify usage
  2. Soft-delete dependencies
  3. Retain recovery window
  4. Purge later
Reversible steps precede irreversible cleanup. Idempotent stages allow a partial deletion to converge on retry.

The problem

Hiding a namespace did not remove the resources behind it. Infrastructure, metadata, dataset topics, and stored blobs could outlive the namespace indefinitely. Manual cleanup was uncommon and risky because the dependencies crossed several systems.

I owned the work to make deletion self-service, complete, and recoverable across the Hollow ecosystem.

The design

I built deletion as a layered cascade: RAW Hollow invokes the Cinder and History workflows, and Cinder deletes its dependent Gutenberg topics and metadata. Each abstraction can use its own lifecycle independently.

A shared asynchronous workflow exposes progress and supports retries. Usage checks refuse deletion when activity cannot be verified. At the data layer, tombstones hide deleted topics while leaving the underlying data intact during the recovery window.

Why soft deletes

One alternative was to copy deleted data into a recovery topic. I favored tombstones because a complete restore also needs retention policies, access control, and pointers to the stored data. Keeping the original state preserved those relationships.

I added restore operations, enforcement across publishing and consuming paths, delayed purging, and routine end-to-end lifecycle tests. I also documented operational recovery so support did not depend on the original implementer.

The result

A single API can complete RAW Hollow teardown in under 30 minutes. Cleaning up stale namespaces reclaimed roughly $270K in annual infrastructure cost.

The recovery path restored needed data after two production deletion incidents. Those cases also exposed opportunities to strengthen the usage guardrails further.

Background & references

Public documentation for the systems discussed here.