Exactly-once-ish: the delivery guarantees we actually promise

Ask a queue vendor about exactly-once delivery and watch the asterisks multiply. Every distributed systems veteran winces at the phrase, because delivery is at-most-once or at-least-once — pick which failure you'd rather have. What you can build is exactly-once effects: the email sends once, the count converges, no matter how many times the message arrives.

The gap between those two sentences is most of reliability engineering. Kilden closes it with three decisions, applied everywhere without exception — and the third one has an ordering subtlety that's easy to get fatally wrong.

Decision 1: at-least-once, honestly

Every consumer follows one iron rule: commit the offset only after the destination write is confirmed. Crash between the write and the commit, and the batch is redelivered — the same event can absolutely arrive twice. That's the deal with at-least-once: duplicates aren't a bug, they're the contract. The alternative (commit first, write after) silently loses events on crash, which is the strictly worse failure for a system whose product is counting things.

So the design question was never "how do we avoid duplicates". It's "who absorbs them" — and the answer has to be whoever writes the effect.

Decision 2: identity is born where retries start

Deduplication needs identity, and identity has to be minted at the source of the retry. Every event gets a UUIDv7 in the client SDK, before the first network attempt — so when a flaky connection makes the SDK resend a batch the server half-processed, the retry carries the same ids and collapses harmlessly.

Events derived from provider webhooks can't do that — the provider controls the payload — so their ids are UUIDv5, deterministically derived from the provider's message id and event type. The provider retries its webhook; we derive the same id; the duplicate vanishes. Same guarantee, no coordination, no negotiation with a third party's retry policy.

From there, each edge absorbs its own duplicates: the ClickHouse events table is a ReplacingMergeTree keyed by that UUID (redelivered batches merge away, counts converge), the messaging side claims uniqueness in Postgres, and outbound webhooks carry idempotency keys so our retries are safe for receivers too.

Decision 3: claim first, send second

The messaging edge deserves the close-up, because an email is the one effect you can't merge away after the fact.

The dispatcher inserts its claim — this journey, this node, one row, unique — before calling the email provider. If the insert conflicts, someone already owns that send: drop it, silently, correctly. If the process crashes after claiming and before sending, the failure is a missing email — visible, retryable, survivable.

Flip the order and the failure flips too: crash after sending but before recording, and the retry sends again. A customer emailed twice by a "reliable" system is the unrecoverable failure; no retry un-sends an email. Choose which failure you'd rather live with, then order your writes so you get that one. And because the campaign graph is a DAG, each node passes at most once per journey — the unique claim isn't fighting the traversal, it's enforcing what the graph already promised.

The honest summary

At-least-once delivery, exactly-once effects, one iron rule per edge: identity minted where retries start, dedup owned by whoever writes the effect, claims ordered before the side effects you can't take back. Nothing here is novel. The entire trick is refusing to make an exception — any exception — for the life of the system.