Measure and learn

Begin with the decision the evidence must inform. Specify the outcome, population, comparison, assignment unit, cost scope, horizon and decision rule before exposure. Then determine whether enough independent observations can mature in time. The measurement design is part of the intervention, not a reporting task added afterwards. [GRW §§37–47; RT §§20–23]

Use random assignment when the treatment, population and sample support it. Assign at account, household or geography level when individual assignment would create contamination. Analyse all assigned eligible units by default, rather than just converters or exposed users. Check assignment integrity, missing outcomes, interference, concurrent changes, outcome delay and the predeclared analysis rule.

For material media, consider a valid platform holdout or geographic design with the method owner. For low-volume B2B, a user-level A/B test may never become informative. Use customer research, larger units, staged changes or qualified observational analysis when appropriate, and state what remains uncertain. Do not withhold an obvious defect repair or required safeguard merely to run an experiment.

Adaptive allocation and causal estimation serve different purposes. Bandits may be appropriate for some mature, frequent decisions but do not replace sample, logging or valid analysis. Do not adopt them as a universal upgrade.

When views disagree, compare definitions, denominators, windows, populations and assumptions. Do not average them into a synthetic truth or pick the most flattering result. MMM requires adequate history and variation plus specialist validation; an available tool is not evidence that the local model is identifiable. [GRW §§44–47]

Illustrative arithmetic only. Assume a valid randomised comparison with the stated eligibility and outcome horizon: 5,000 eligible accounts receive a treatment and 5,000 receive the comparator. At the fixed horizon, 4,550 treated and 4,450 control accounts remain. Retention is 91% versus 89%: an estimated 2 percentage-point effect.

That implies 100 additional retained accounts among the 5,000 treated. If their additional contribution over the stated follow-up horizon is CU300 each, the estimated contribution before treatment cost is CU30,000. If CU28,000 covers the actual 5,000-treatment deployment and is excluded from that contribution, the point-estimate net is CU2,000.

A future rollout treating 10,000 comparable accounts projects 200 additional retained accounts and CU60,000 before treatment cost only if the effect transports. Its costs must be priced for that rollout. Neither calculation demonstrates statistical precision or justifies scale without uncertainty and guardrails.

Editorial Resolution ER08: GRW §41 moves from the original treatment group to a per-10,000 projection without a clear cost basis. This example separates the trial estimate from the rollout and labels the cost assumption. Prefer direct contribution per assigned account when it can be observed, reducing reliance on a value-per-retained-customer approximation.

T3 stores a fact, finding, hypothesis, decision or learning with its source, population, method, outcome, uncertainty, owner and review trigger. Keep nulls, contradictions, cancellations and invalidated tests. A human method owner approves evidence promotion; a summarising model cannot make its own interpretation authoritative. [SYS §9; OPS §38]

A useful learning statement says: “For this population, under this treatment and horizon, the evidence supports this conclusion within these limits; use it for this decision until these conditions change.” Review when product, price, customer mix, channel or policy shifts. Start with linked records and ordinary search. Measure whether subsequent decisions actually use them before adding specialised memory technology.