Separate campaign credit from additional impact
A customer sees a campaign and later buys. Attribution assigns credit under a reporting rule, such as a click window. Incremental measurement asks a different question: how would outcomes have differed without the campaign? A purchase can receive campaign credit even when the customer would have bought anyway.
Google’s Conversion Lift documentation makes this distinction explicitly. Its product implementation has its own eligibility and methodology; the general lesson for a CRM team is to state the counterfactual question before choosing the report. A well-defined randomized holdout can provide a practical comparison.
Write the experiment contract first
Specify the eligible population, the unit of randomization, the proposed intervention and the control experience. Define one primary outcome, the observation window, cost measures and guardrails such as complaints or returns. Document the minimum effect that would change a business decision and plan the sample and duration around it.
For a second-purchase test, the unit might be the resolved customer and the primary outcome a second valid order within thirty days of assignment. Both groups should continue receiving necessary service messages. The withheld element is the campaign being tested, not essential delivery or support communication.
Randomize before activation and preserve assignment
Assign eligible units randomly and keep the assignment stable. The randomization unit should match the decision: shared accounts, households or organizations may require a design that prevents one unit’s treatment from affecting another. NIST’s discussion of randomized designs explains the underlying principle of assigning factor levels to experimental units randomly.
Store the assignment time, eligibility definition, experiment version and intended action. Check group sizes and pre-treatment characteristics for obvious implementation problems. Do not move people between groups because they engage or purchase. Their later behavior is part of the outcome.
Use comparable rates and a stable denominator
The following numbers are fictional. Suppose 10,000 customers are assigned to the campaign and 10,000 to a holdout. Within the same window, 1,200 campaign-assigned customers and 1,000 holdout customers make a qualifying purchase. The rates are 12% and 10%. The observed absolute difference is 2 percentage points, and relative lift is 20%.
Applied to the 10,000 campaign-assigned customers, the rate difference corresponds to an estimated 200 additional purchasers. It does not imply that all 1,200 purchases were caused by the campaign, nor identify which individual purchases were incremental. With unequal group sizes, compare rates and scale the control expectation to the treatment population.
| Group or measure | Assigned customers | Purchasers | Purchase rate |
|---|---|---|---|
| Campaign | 10,000 | 1,200 | 12% |
| Holdout | 10,000 | 1,000 | 10% |
| Observed difference | — | 200 estimated additional purchasers | 2 percentage points |
Report uncertainty alongside the estimate
Under a simple independent two-proportion normal approximation, the fictional example’s 95% confidence interval for the rate difference is approximately 1.13 to 2.87 percentage points. This calculation assumes the design supports independent units and adequate sample sizes. Clustered assignments or more complex designs need an analysis that respects that structure.
A confidence interval describes the uncertainty of the estimation procedure; it is not a guarantee that future campaigns will repeat the result. When an interval includes zero, the data may be compatible with a loss as well as a gain. An inconclusive result is different from proving that the campaign has no effect.
Choose the analysis and stopping rule before launch. Repeatedly checking results and stopping at the first attractive number changes the error properties of a conventional fixed-horizon test. Use the planned end point or a properly designed sequential method, and account for multiple comparisons where relevant.
Protect the comparison during execution
A holdout customer who receives the same offer through another journey can weaken the contrast. Record exposure across channels, define which other campaigns continue and monitor contamination. Identity changes and repeated assignment can also corrupt the comparison. Keep one durable experiment identity wherever possible.
For the primary intention-to-treat comparison, keep everyone as originally assigned, including failed deliveries and unopened messages. A comparison restricted to openers answers a different, selected-population question. Report delivery quality separately so the team can distinguish an execution problem from a weak policy.
Translate lift into a decision
Additional purchases are one part of value. Compare net contribution per assigned customer after product costs, returns, incentives and relevant variable delivery costs. The most profitable policy may not have the largest purchase-rate lift. Use the same accounting definition in both groups and retain longer-term indicators when the decision requires them.
In Binoban, customer definitions, audiences, events and analytics provide parts of the operating context. Confirm the assignment, holdout enforcement and reporting capabilities available in the deployment before running the design. The calculations in this article are an explanatory example, not a screenshot or claim of a native automatic lift report.
A reviewable result includes the eligible population, assignment rule, outcome definition, observation window, group counts, rates, estimated difference, uncertainty and cost interpretation. Together, these let the business decide whether to expand, revise or stop the policy with a clear understanding of what the evidence supports.
Further reading
Take the next step