Milestone map
Milestone map
3 milestones
Design and run your first A/B test
2–4 weeks
Design an A/B test on a real product, page or email. Write a hypothesis and pick one primary metric. Calculate the sample size before launch and don't peek early. Deciding the rules before the data arrives is what makes a result believable.
Proof required
Submit the pre-launch plan (hypothesis, metric, sample size and stopping rule) with its date, and the result with the numbers.
What gets checked
- Plan dated before the test started
- Sample size calculated, not guessed
- Stopped at the planned sample, not early
Common mistakes
- Deciding the metric after seeing results
- Stopping when it looks significant
- Too few users to detect anything
Powstik Guide
A pre-planned A/B test on a real audience.
Steps
- Write the hypothesis and one primary metric.
- Estimate baseline rate and minimum effect you care about.
- Calculate the sample size and fix it.
- Launch, and don't check results until you reach it.
- Report the result with the numbers.
Template
Test plan (date: ) Hypothesis: Primary metric: Baseline rate: Minimum detectable effect: Sample size per arm: Stopping rule: Result Control: n= rate= Variant: n= rate= Difference and p-value:
What gets sent back
- A plan written after the test.
- Early stopping.
- No sample size calculation.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Check the plan date against the test dates
- Recompute the sample size
- Check the stopping rule was followed
You'll sign in first, then come straight back here.
Run two more tests and report effect sizes
3–6 weeks
Run two more tests with the same discipline. Report effect sizes and confidence intervals, not just significance. Include any test that found nothing; null results are results.
Proof required
Submit plans and results for two more tests, each with effect size and confidence interval, including any null result.
What gets checked
- Effect size and interval reported for each
- Null results reported, not hidden
- Each test pre-planned like the first
Common mistakes
- Only reporting wins
- p-values without effect sizes
- Changing the variant mid-test
Powstik Guide
Two more disciplined tests with effect sizes.
Steps
- Reuse your plan template for each test.
- Keep a log of every test you start.
- Compute effect size and a 95% interval.
- Report nulls with what they rule out.
Template
| Test | Hypothesis | n per arm | Effect size | 95% CI | Decision | |------|------------|-----------|-------------|--------|----------|
What gets sent back
- Hidden null results.
- No confidence intervals.
- Variants changed mid-test.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Check intervals are reported
- Check all tests run are listed
- Check plan dates
You'll sign in first, then come straight back here.
Turn results into decisions, reviewed
1–2 weeks
Write up what decision each test drove and defend it to an experienced analyst. They should challenge your stats and your business reasoning. A test is only worth running if it changes what someone does.
Proof required
Submit a decisions memo for all three tests and review notes (250+ words) with the analyst's background and their challenges.
What gets checked
- Each test tied to a real decision taken
- Reviewer has run experiments professionally
- At least one challenge changed your view
Common mistakes
- Decisions not actually taken
- A reviewer without experimentation experience
- No change after review
Powstik Guide
A decisions memo, challenged by an analyst.
Steps
- For each test, write the decision it drove.
- Note who made the decision and when.
- Book an analyst or data scientist who runs experiments.
- Ask for challenges on stats and on the decision.
Template
| Test | Result | Decision taken | Owner | Date | |------|--------|----------------|-------|------| Review session log Date: Duration: Reviewer name: Reviewer role and experience: Challenge 1 (a statistical or business choice): What they asked: My answer: Challenge 2: What they asked: My answer: Change I made after the session:
What gets sent back
- Decisions never taken.
- A reviewer without experiment experience.
- No change after review.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Confirm decisions were implemented
- Confirm reviewer background
- Check the change after review
You'll sign in first, then come straight back here.