Paid Acquisition Retention: An Experiment Design
How do I test whether a retention tactic works?
Assign eligible first-time purchasers persistently to treatment or control and hold every other term invariant, then read contribution rather than engagement. Persistence matters because customers reassigned between cohorts contaminate both arms and produce a result that cannot be attributed to the intervention.
For the surrounding account decisions, compare Paid-Social Landing Pages: A Controlled Experiment and use Paid Acquisition Cohort Value: A Realized Model as the next diagnostic.
In short
This guide owns one decision artifact: the filled, auditable structure below. Reader-supplied thresholds stay explicit; missing evidence stays missing.
Post-purchase replenishment-reminder experiment
- Cell A: Control: eligible first-time purchasers receive the existing post-purchase service communications only.
- Cell B: Treatment: the same service communications plus one named replenishment reminder at the reader-selected product-age point, with identical terms and opt-out.
- Falsifiable expectation: The reminder changes matured contribution per assigned customer and the preregistered repeat-purchase rate; zero/reversal falsifies the direction.
- Held invariant: Eligible acquisition cohort, product/SKU, initial offer, service messages, price, fulfilment, measurement, assignment unit, and analysis rules. The reminder is the treatment, not a controlled variable.
- Budget allocation: Declare contact/exposure budget and maximum eligible contacts before randomization; split eligible customers 1:1 with a persistent assignment key.
- Maturity window: Follow every assigned customer through the reader-selected replenishment horizon plus cancellation/refund cutoff; younger cohorts remain ineligible for the read.
- One primary metric: Matured contribution per assigned customer = (recognized cohort revenue − product, fulfilment, payment, refund, service, reminder, and acquisition costs) ÷ (eligible assigned customers). Repeat-purchase rate = customers with a recognized repeat order ÷ eligible assigned customers. Source: assignment, messaging, customer, and order/cost ledgers.
- Stop rule: Stop for consent, unsubscribe, claim, deliverability, inventory, margin, or customer-harm threshold.
- Inconclusive rule: Inconclusive when planned power is unmet, cross-cell message contamination exceeds tolerance, assignment IDs fail, exposure delivery is materially imbalanced, or the cohort has not matured.
Interpretation boundary
Use the randomized replenishment reminder only for its stated decision. Assign eligible first-time purchasers persistently to existing service messages or those messages plus one timed replenishment reminder. Insufficient power, failed assignment, exposure imbalance, message contamination, or cohorts below the replenishment/refund horizon makes the result inconclusive. Reader-supplied thresholds remain inputs, not universal standards.
Can software help?
Bach.ai audits your connected Meta account, estimates the revenue impact of what it finds, and proposes specific fixes. It applies a change only after you approve it. Think of it as an automated audit layer that surfaces issues and proposed fixes for your review — not a replacement for your team’s judgment, and creative production is not its core job, though the Pro and Agency plans can generate a limited number of variants.
FAQ
How do you randomize a post-purchase replenishment reminder test?
Assign eligible first-time purchasers persistently to existing service messages or those messages plus one timed replenishment reminder.
Which failures make a paid-acquisition retention test inconclusive?
Insufficient power, failed assignment, exposure imbalance, message contamination, or cohorts below the replenishment/refund horizon makes the result inconclusive.
When does a retention experiment support a causal conclusion?
It compares the declared cells in the randomized replenishment reminder. A causal interpretation additionally depends on valid assignment, stable invariants, adequate power, and contamination within the preregistered limit.
Method and sources
“Assign eligible first-time purchasers persistently to treatment or control and hold every other term invariant, then read contribution rather than engagement.”
Source: Where this guide describes platform behaviour, it follows Meta’s published advertising and Marketing API documentation, which changes without notice — verify anything load-bearing against the current version before you act on it. Every threshold the guide asks you to supply is first-party, drawn from your own account exports and commerce ledger, because no external benchmark can stand in for your own margin structure.