Paid Acquisition Retention: An Experiment Design

For the surrounding account decisions, compare Paid-Social Landing Pages: A Controlled Experiment and use Paid Acquisition Cohort Value: A Realized Model as the next diagnostic.

In short

This guide owns one decision artifact: the filled, auditable structure below. Reader-supplied thresholds stay explicit; missing evidence stays missing.

Post-purchase replenishment-reminder experiment

• Cell A: Control: eligible first-time purchasers receive the existing post-purchase service communications only.
• Cell B: Treatment: the same service communications plus one named replenishment reminder at the reader-selected product-age point, with identical terms and opt-out.
• Falsifiable expectation: The reminder changes matured contribution per assigned customer and the preregistered repeat-purchase rate; zero/reversal falsifies the direction.
• Held invariant: Eligible acquisition cohort, product/SKU, initial offer, service messages, price, fulfilment, measurement, assignment unit, and analysis rules. The reminder is the treatment, not a controlled variable.
• Budget allocation: Declare contact/exposure budget and maximum eligible contacts before randomization; split eligible customers 1:1 with a persistent assignment key.
• Maturity window: Follow every assigned customer through the reader-selected replenishment horizon plus cancellation/refund cutoff; younger cohorts remain ineligible for the read.
• One primary metric: Matured contribution per assigned customer = (recognized cohort revenue − product, fulfilment, payment, refund, service, reminder, and acquisition costs) ÷ (eligible assigned customers). Repeat-purchase rate = customers with a recognized repeat order ÷ eligible assigned customers. Source: assignment, messaging, customer, and order/cost ledgers.
• Stop rule: Stop for consent, unsubscribe, claim, deliverability, inventory, margin, or customer-harm threshold.
• Inconclusive rule: Inconclusive when planned power is unmet, cross-cell message contamination exceeds tolerance, assignment IDs fail, exposure delivery is materially imbalanced, or the cohort has not matured.

Interpretation boundary

Use the randomized replenishment reminder only for its stated decision. Assign eligible first-time purchasers persistently to existing service messages or those messages plus one timed replenishment reminder. Insufficient power, failed assignment, exposure imbalance, message contamination, or cohorts below the replenishment/refund horizon makes the result inconclusive. Reader-supplied thresholds remain inputs, not universal standards.

Can software help?

Bach.ai audits your connected Meta account against 100+ checks, ranks what it finds by estimated impact, and proposes specific fixes. It stays read-only until you approve a change, then executes the approved change on Meta; connected Google Ads data is used for intelligence only. Think of it as an automated audit layer that surfaces issues and proposed fixes for your review — not a replacement for your team’s judgment, and it does not generate your creative.

FAQ

How do you randomize a post-purchase replenishment reminder test?

Assign eligible first-time purchasers persistently to existing service messages or those messages plus one timed replenishment reminder.

Which failures make a paid-acquisition retention test inconclusive?

Insufficient power, failed assignment, exposure imbalance, message contamination, or cohorts below the replenishment/refund horizon makes the result inconclusive.

When does a retention experiment support a causal conclusion?

It compares the declared cells in the randomized replenishment reminder. A causal interpretation additionally depends on valid assignment, stable invariants, adequate power, and contamination within the preregistered limit.