Kafka never carries blobs: session replay as a claim check

For a while, some of our session replays played back as minutes of black screen. The timeline insisted something had been recorded. The chunks were all in storage, checksums intact. Nothing was corrupted, nothing was lost — and the replays were still unwatchable.

The bug was in a filename. We'll get there — but first, the architecture it lived in, because the bug only makes sense once you see what replay's cargo actually looks like.

The heaviest thing on the pipeline

Session replay records the DOM and every mutation after it; a busy session produces megabytes. The rest of Kilden moves events of a few kilobytes through Kafka — replay's chunks would have turned the same brokers into a reluctant blob store, with broker memory, replication traffic and retention all sized for the biggest cargo instead of the common case.

So replay follows one hard rule inherited from the pipeline's design: Kafka never carries blobs. It carries claims about blobs.

The recorder ships compressed chunks — gzip, as the SDK sent them, up to 4MB — to a dedicated ingest endpoint that writes them straight to object storage:

{project_id}/{session_id}/{recording_id}/{chunk_index}.json.gz

What goes to Kafka is a pointer of roughly 200 bytes: session, recording, project, person, chunk URL, index, byte size, time range, page URL, and whether the chunk saw an error. A consumer batch-inserts those pointers into the session index in ClickHouse — commit after insert, dead-letter queue for poisoned pointers, exactly like every other consumer in the system. The pointer topic gets Kafka's ordering and replayability; the blobs get object storage's economics; and a 30-day lifecycle rule on the bucket does retention by itself.

The filename that ate our replays

Now, that recording_id in the storage key. It wasn't there at first. Look at what happens without it:

chunk_index counts from 0 — per page load. But a session lasts as long as the user keeps coming back within 30 minutes. Visit a page, record chunks 0–4, click a link — and the new page starts writing chunk 0 again. Same session, same key prefix, same filenames. The second page load silently overwrites the first one's chunks, including chunk 0, which holds rrweb's FullSnapshot: the single chunk playback cannot start without.

Hence the black screens. The player had events — mutations, clicks, scrolls — but the snapshot they mutated from had been overwritten by a different page's opening frame. Nothing in the write path failed; every PUT succeeded. That's what made it maddening: the bug produced no errors, only silence where a replay should be.

The fix is the key: each recorder start mints a recording_id (a UUIDv7, validated at ingest because it lands in a storage path), and both the storage key and the index's sorting key include it. Two recordings of one session now coexist instead of overwriting, while redelivered pointers still dedup exactly as before. The player merges recordings by timestamp and skips any prefix it can't reproduce.

The rule worth stealing

A storage key must contain every axis on which writes can repeat. Session and index weren't enough axes — the index could reset while the session lived on, and any key where one part can reset while another survives is a collision waiting for its moment. If you take one thing from this post, audit your keys for that pattern. Ours passed review, passed tests, and shipped; the only thing that caught it was a user coming back for a second look.