Experiments (A/B testing)

An experiment in Kilden is not a second traffic splitter. It is a lens over a multivariate feature flag you already have: the flag keeps deciding who sees what, exactly as it did before, and the experiment reads the exposures your SDK is already sending and tells you whether the difference is real.

That has two consequences worth knowing up front. Starting an experiment changes nothing about what your users see. And any flag with two or more variants is already measurable — you do not have to instrument anything new.

Before you start

An experiment needs:

• A multivariate flag with at least two variants, one of them marked as the control.
• Your code actually asking for the flag. Exposures come from getFeatureFlag() / isFeatureEnabled(). If your code branches without calling one of them, the experiment measures nothing — and says so.
• A goal event: the thing you want more of. Conversion (did this person do it at least once?) or a value per person (a sum, or a count).

// This call is what puts the person in the experiment.
const variant = await kilden.getFeatureFlag('checkout_flow');

if (variant === 'one_step') {
  renderOneStepCheckout();
} else {
  renderClassicCheckout();
}

The SDK sends one exposure per flag per session; the analysis keeps the first variant each person saw.

Creating one

Experiments → New experiment, in three steps: the hypothesis, the flag and its control, and what counts as success.

The last step estimates how long the test needs, using the flag’s own recent traffic. If the answer is “eight months”, it says so before you start rather than after — the most common way an A/B test fails is being unable to detect the effect it was looking for.

While an experiment runs, the flag’s split is frozen. Changing weights mid-flight mixes two different assignment configurations under one result, which is the quietest way to ruin a test. Pause the experiment if you need to edit the flag.

Reading the results

The page opens with a verdict in words, and everything below it exists so you can audit that verdict.

Why a probability and not a p-value

Kilden reports “B beats the control with 91% probability”. That is a bayesian statement about this experiment, and it means what it sounds like it means.

A p-value does not. It answers a different question (“how surprising would this data be if there were no difference at all?”), it is routinely misread as the probability of being right, and it punishes exactly the behaviour every real team has — looking at the results every day. Under continuous observation, a p-value based decision rule inflates false positives badly. The bayesian reading degrades far more gracefully.

Alongside it you get:

• The credible interval — the range the true effect plausibly lives in. Kilden never shows an uplift without one.
• Expected loss — how much you give up on average if you pick this variant and it turns out you were wrong. This is the number that actually answers “is it safe to ship?”.

The effect chart

One row per variant, showing the uplift against the control with its 95% credible interval. Zero is always in the frame, and an interval that crosses zero is drawn in neutral grey — because an effect that might be positive and might be negative is not a result yet.

It is deliberately not a bar chart of conversion rates: bars invite a truncated axis that turns 2.1% vs 2.3% into a mountain.

The posterior curves

The distributions of what each variant’s true rate might be. The overlap is the uncertainty: two curves sitting on top of each other mean you cannot tell them apart yet, no matter what the leading number says.

Evolution

The probability of beating the control, day by day, computed with only what was known on each day. Watching that line cross 95% early and then fall back is the clearest argument there is against stopping the moment a result looks good.

The validity diagnostics

Three checks run on every experiment. A result that fails them is worse than no result, because it looks just as convincing — so Kilden does not report a winner when they fail, it does not soften the reading.

Split (SRM)

Compares the traffic each arm actually received against what the flag was configured to send. A mismatch means something broke assignment: an asymmetric bot filter, a weight change mid-flight, or an integration that only asks about the flag on one branch. Any conclusion drawn on top of that is noise.

Contamination

People who saw more than one variant. This usually happens when anonymous visitors identify themselves mid-test: the bucket depends on the ID, so the ID changing can move someone across arms.

Kilden excludes those people from the analysis and reports the rate instead of silently correcting it. Over 2%, the result is marked as compromised. The exclusion has a small bias of its own — it drops people who logged in during the test — and that is a smaller, visible problem than an invisible one.

Sample

Every arm needs at least 100 people before any inference is reported, and the verdict tracks progress against the sample your MDE implies.

This is where you will meet the difference between leading and decisive. An arm can cross 95% on day three with a quarter of the sample: real effects and pure noise both do that. Kilden calls it leading, tells you how much sample is missing, and still lets you conclude — the decision is yours, the tool just refuses to pretend the test is over.

Concluding

Concluding freezes the results exactly as they stand, as the evidence behind the decision, together with whatever you write down about why you decided it. That snapshot is never recomputed: the data underneath moves — merges, retention, backfills — and a number that changes after the decision is not evidence.

Concluding does not change your flag. Rolling the winner out to 100% is a separate, explicit action, with the flag’s own audit trail. Kilden never changes what your product does on its own.

Limits in v1

• Browser traffic only. Server-side SDKs do not emit exposure events yet, by design.
• One goal event as the primary metric; multi-step funnels are approximated with the last step.
• Revenue metrics are winsorized at the 99th percentile before comparison, so one enormous order cannot move the mean. The page tells you how many values that touched.