One door in: why every event takes the same path

Ask us for the exact bug and we'll disappoint you. After fifteen years of building products — MVPs shipped fast, technical debt taken on knowingly, most of it repaid, some of it definitely not — the individual incidents blur together. What survives is the pattern. And the pattern is this: every data disaster we can half-remember started with data entering a system through more than one door.

We don't remember the projects anymore. We remember the shapes.

The script that skipped the logic. A backfill or a manual fix that inserted rows straight into the database. It ran perfectly. It also skipped every validation and side effect the application would have run — welcome emails that never went out, caches that never got invalidated — and left behind rows the rest of the code was never taught to believe in.

The "just for now" insert. Written in week two of an MVP, when the real write path was one INSERT away and the deadline was closer. Perfectly reasonable at the time — that's what makes early technical debt debt and not malpractice. Years later a new feature assumed a field that shortcut never wrote, and it failed only for the rows that had come in through the side door. Those are the worst bugs: rare, data-dependent, invisible in staging.

The impossible state. A user who updated their profile before registering. Not clock skew, not a logging artifact — the update genuinely got processed before the create, because the two events traveled different paths and nobody had ever promised they'd arrive in order.

That last one deserves its own section, because we learned it the way it's traditionally taught: the hard way, in production.

The ordering lesson

Kafka makes you a promise, and it's much narrower than people assume. Between two topics there is no ordering guarantee at all. Between two partitions of the same topic, none either. The only ordering that exists is within a single partition — which means the only ordering you get is the ordering you design for when you choose the partition key.

We didn't read that in the documentation and nod sagely. We lived the version where a user registers and immediately updates their profile — as new users do; it's practically the first thing they do — and the profile update arrives before the registration. Multiply that by a fleet of microservices, each consuming from its own topics at its own pace, and you end up defending yourself with locks: SELECT FOR UPDATE here, advisory locks there, a distributed mutex somewhere one really shouldn't be. The system gets slower and it still breaks, because locks can serialize what arrives — they can't reorder it.

Starting from the scar tissue

When we started Kilden we knew one thing before we knew almost anything else: ingestion was the core. A customer data platform computes practically everything — analytics, campaigns, replays, flags — from the event stream, so the hot path had to be fast, minimal, and boring. That's why it's Go. And every extra process on that path, every duplicated write, every "internal fast lane" is another place for the old scars to reopen.

So we iterated on the architecture diagram until a rule emerged that made all three shapes of pain structurally impossible, and then we wrote it down as law: one way in. Events from the web SDK, events from server SDKs, email opens arriving as provider webhooks, even the events our own panel emits about itself — every one of them walks through the same capture endpoint into the same raw events topic.

Not two ways. Not "one way plus a shortcut for internal stuff". The shortcut is the thing we were there to ban.

What the single door buys

Everything between the door and the store happens exactly once, for every event, in one place:

• Dedup — every event carries a client-generated UUID; retries are idempotent no matter which producer sent them.
• Identity resolution — the enricher resolves distinct_id → person, applies merges, and stamps the result. A webhook-derived event gets the same treatment as a browser click.
• Verification — the trust level of every event is computed at one choke point, not re-implemented per source.
• Fan-out — analytics, campaigns, and the live tail all consume the same enriched stream. There is no "the campaign engine saw an event analytics didn't".

The email feedback loop is where this gets fun. Kilden's dispatcher sends a campaign email; the provider reports the open via webhook; that webhook becomes a $email_opened event that re-enters through capture like any other event. Which means campaigns can trigger on it, analytics can chart it, and the live tail shows it — all for free, because nothing downstream knows or cares that this event was born from an email instead of a click.

Partition keys do the locking

Remember the locks? Here's what the same problem looks like when you design for it instead of defending against it.

The raw topic is keyed by distinct_id; the enricher re-keys the enriched topic by person_id. That re-keying is the whole concurrency story:

• Same distinct_id → same partition → identity resolution for one visitor is single-threaded by construction. The register-then-update race that haunted us for years loses its teeth: both events land in the same partition, so one consumer processes them one at a time, in one agreed order — there is no second path for the update to race through while the create is still in flight. And because partitioning orders the partition, not the network (a retrying producer can still interleave appends), the enricher doesn't even assume the create comes first: the first sight of any distinct_id creates the person, so an update with no create in front of it just creates the person and moves on.
• Same person_id → same partition → the campaign engine's per-person state machine advances one event at a time, in order, without coordination.

The hot path holds no locks — not because we were disciplined about locking, but because the partitioner already serialized everything that needed serializing. When people ask what the event streaming layer buys us, this is the answer — not throughput, ordering. Throughput you can buy; per-person ordering without locks you have to design for.

The failure story

One door also means one failure story, told once. No consumer of ours lets a malformed event block its partition: each one retries a few times, then parks the event in the topic's dead-letter queue and the stream moves on. Offsets are committed only after the destination write is confirmed — so the guarantee everywhere is at-least-once, with dedup at the edges absorbing the duplicates that guarantee implies.

None of this required cleverness. It required taking fifteen years of the same bug in different costumes seriously enough to make it unrepresentable — saying no to one shortcut, every time, for the life of the system. That's cheaper than it sounds. The alternative is a price we'd already paid too many times to pay again.