Recipe: Holdout group experiment design

1. Define your holdout percentage

Decide what fraction of users will never see the feature. Common choices are 5%, 10%, or 20%. Larger holdouts give more statistical power but reduce the rollout audience.

2. Create the feature flag

In the Meridian dashboard, create a boolean flag named holdout_experiment_v1. Set the default rule to true for the treatment group and add an override rule that forces false for the holdout percentage.

3. Instrument your application

Wrap the feature behind a single conditional. Log an exposure event so Meridian can track who entered the experiment.

if (meridian.isEnabled("holdout_experiment_v1")) {
  showNewCheckout();
  meridian.track("holdout_exposed");
}

4. Define success metrics

Pick a primary metric (e.g. conversion rate, revenue per user) and one guardrail metric (e.g. error rate). Meridian computes lift, confidence intervals, and p-values automatically.

5. Run and monitor

Let the experiment run for at least one full business cycle. Meridian will alert you if the guardrail metric degrades significantly. Do not peek at results before the minimum sample size is reached.

6. Ship or roll back

If the treatment wins with statistical significance and no guardrail regression, ramp the flag to 100%. Otherwise, kill the flag and document what you learned.