How to Run an A/B Test: A 6-Week Roadmap From Hypothesis to Decision

9 min read ยท 2026-10-12

To run an A/B test, write one hypothesis and pick one primary metric. Calculate how many users you need, then split traffic at random between the current version (A) and the new one (B). Run the test for full weeks until you reach that sample size. Only then compare the results and decide. A simple first test takes about 6 weeks from idea to decision. The test itself takes 2 of those weeks, and the rest goes to preparation and analysis.

Most guides on how to run an a/b test list seven steps and stop there. This guide turns those steps into a dated roadmap, with phases, concrete tasks and a milestone you can check at the end of each phase. It works for a signup page, a pricing page, an onboarding flow or an email. It suits product managers, founders and marketers who run experiments without a full data team.

The roadmap at a glance

Goal: Run one clean A/B test and reach a ship, drop or retest decision that the team trusts. Duration: 6 weeks, with a 2-week test

  1. Frame the Problem and the Hypothesis (Week 1)

    Choose one problem that matters and turn it into a statement the test can prove wrong.

    • Pull the last 4 to 8 weeks of data for the page or flow and find the step where most users drop off.
    • Collect the evidence behind your idea: session recordings, support tickets, user interviews, survey answers.
    • Write the hypothesis in this form: "If we change X for audience Y, metric Z will go up, because of reason R."
    • List the one change version B will contain, and nothing else.

    Milestone: A one-page test brief with the hypothesis, the change and the evidence, approved by the product owner.

  2. Choose Metrics and Size the Test (Week 2)

    Decide what success means and how much traffic you need before anyone sees the results.

    • Pick one primary metric that the change can move directly, such as signup completion rate.
    • Add 2 or 3 guardrail metrics that must not get worse, such as refund requests, page load time or unsubscribes.
    • Use a sample size calculator with your current conversion rate and the smallest lift worth shipping, to get the number of users per variant.
    • Divide that number by your weekly traffic to get the run time, rounded up to full weeks.
    • Write the decision rule now: what result means ship, what means drop, what means retest.

    Milestone: The test brief shows the primary metric, the guardrails, the sample size per variant, the planned end date and the decision rule.

  3. Build and QA the Variant (Week 3)

    Make sure version B works and that users are split fairly.

    • Build version B in your testing tool, feature flag system or email platform.
    • Set a 50/50 random split at the user level, not the session level, so the same person always sees the same version.
    • Test both versions on desktop, mobile and the main browsers, logged in and logged out.
    • Check that tracking fires correctly for the primary metric and every guardrail in both versions.
    • Run a short internal test and check that each event shows up in your analytics.

    Milestone: QA checklist signed off, with both variants rendering and every event recorded for both.

  4. Run the Test (Weeks 4-5)

    Collect clean data without touching the test.

    • Launch at the start of a week so the test covers full weekly cycles.
    • If the Week 2 calculation asks for more than 2 weeks, extend this phase by the same number of weeks.
    • On day 2, run a sample ratio check: if you planned 50/50, the counts in A and B should be close. A large gap means the split or the tracking is broken.
    • Monitor guardrails daily. Stop early only if something is broken or a guardrail is clearly getting worse.
    • Freeze other changes to the same page or flow until the test ends.
    • Do not read the primary metric to pick a winner before the planned end date.

    Milestone: The planned sample size is reached on the planned end date, and the sample ratio check passes.

  5. Analyze and Decide (Week 6, first half)

    Compare A and B honestly and apply the rule you wrote in Week 2.

    • Calculate the difference in the primary metric with its confidence interval, not only the p-value.
    • Check every guardrail metric for a meaningful drop.
    • Look at the main segments you listed in advance, such as new vs returning users or mobile vs desktop, without hunting for new ones.
    • Apply the decision rule: ship, drop or retest with a bigger sample.

    Milestone: A written decision with the numbers behind it, shared with the team.

  6. Ship and Document (Week 6, second half)

    Turn the result into a change in the product and a lesson the team can reuse.

    • Roll out the winning version to all users, or remove version B.
    • Add a short summary to your experiment log: hypothesis, result, decision, what you learned.
    • Write the next hypothesis this result suggests and add it to the backlog.
    • After the 6 weeks, watch the primary metric for 2 more weeks after rollout to confirm the effect holds.

    Milestone: The experiment log entry is published and the next test is in the backlog.

How to Write a Hypothesis Your Test Can Actually Prove Wrong

A good hypothesis names one change, one audience, one metric and one reason. "Make the pricing page better" can't be tested. "If we show the annual price first to visitors from the pricing link in our emails, upgrade clicks will go up, because interviews showed people compare total cost" can be tested. When the test ends, the result either supports it or doesn't.

The reason part matters more than most teams think. If the test fails, the reason tells you what you got wrong about your users, and the next test starts from there. Without it, a losing test teaches you nothing.

Here are three examples you can adapt:

  • Signup: If we cut the signup form from 6 fields to 3, completion rate will go up, because recordings show people leave at the company size field.
  • Onboarding: If we show a sample project on first login, the share of new users who create their own project in week one will go up, because users say they don't know where to start.
  • Email: If the subject line names the feature instead of the release number, open rate will go up, because past emails with concrete subjects did better.

Which Metrics to Track: Primary, Guardrail and Sanity

Every test needs three kinds of metrics, and each one does a different job. Pick them before launch, never after.

If you track twenty metrics and pick the best-looking one at the end, you will find a "winner" by chance. One primary metric, decided in advance, protects you from that.

  • Primary metric: the one number that decides the test. It should sit close to the change. A button change should be judged on clicks or completions, not on yearly revenue.
  • Guardrail metrics: numbers that must not get worse, such as revenue per visitor, error rate, load time, refunds or support tickets. A higher signup rate with a jump in refunds is not a win.
  • Sanity metrics: checks that the test itself works, such as the sample ratio between A and B and the number of users with tracking events.

How Long Should an A/B Test Run?

An A/B test should run until it reaches the sample size you calculated, and for at least one full week. Two full weeks is safer, because it covers weekday and weekend behavior twice. That is why this roadmap plans a 2-week test. The sample size depends on three inputs: your current conversion rate, the smallest lift you care about and the confidence level you choose. Common defaults are 95% confidence and 80% power.

If the Week 2 calculation asks for more than 2 weeks, extend the "Run the test" phase by the extra weeks. If it asks for more than about 4 weeks of traffic, don't run the test anyway and hope. Test a bigger change that could produce a bigger effect. You can also move the test higher in the funnel, where traffic is larger, or switch to qualitative research with user tests.

Stopping early is the most common way teams fool themselves. Results swing a lot in the first days. If you check every morning and stop the first time B looks ahead, you will ship changes that do nothing. Set the end date in Week 2 and stick to it.

How to Read the Results and Make the Call

Every test ends in one of three outcomes, and your Week 2 decision rule already says what to do with each.

A flat result is still useful. It tells you that this lever doesn't move this metric much, so you can spend your next test somewhere else. Log it like any other result.

  • Clear win: the primary metric improved, the confidence interval does not cross zero, and no guardrail got worse. Ship version B.
  • Clear loss or harm: the primary metric dropped or a guardrail got worse. Keep version A and write down why the reason in your hypothesis was wrong.
  • Flat or unclear: the interval includes zero. The change probably has no effect large enough to matter. Keep the simpler version. Retest only if you have a reason to expect a larger effect with more data.

Fitting A/B Tests Into Your Product Roadmap

One test takes about 6 weeks from start to finish. The phases can overlap, though: you can frame the next test while the current one runs. A practical rhythm for a small team is one test running at a time on each page or flow, with the next one ready to launch the week the current one ends.

Put tests on your roadmap as their own items, with the hypothesis as the title and the decision date as the milestone. This shows stakeholders that a feature is waiting on evidence, not stuck. It also stops two tests from running on the same page at once and spoiling each other's results. If you are new to building one, start with a simple method for making a roadmap, or pick a layout from a set of roadmap templates. For a bigger release, run a beta test first to catch problems before you measure anything. You can also check your plan against the most common roadmap mistakes.

Common mistakes to avoid

  • Stopping the test as soon as B looks ahead; set the end date from the sample size calculation, and only stop early for broken tracking or a guardrail problem.
  • Testing five changes in one variant; change one thing, so you know what caused the result.
  • Choosing the success metric after seeing the data; write the primary metric and decision rule into the brief before launch.
  • Skipping the sample ratio check; compare the user counts in A and B on day 2, and fix the split before trusting any result.
  • Running a test on a page with too little traffic; check the run time in Week 2. If it goes past about 4 weeks, test a bigger change or a busier page.
  • Forgetting to log flat and losing tests; record every result, so the team doesn't repeat ideas that already failed.

Frequently asked questions

How many users do I need for an A/B test?

It depends on your current conversion rate and the smallest lift you want to detect. Enter both into a sample size calculator to get the number of users per variant. Low conversion rates and small expected lifts need much more traffic. If the number is out of reach, test a bigger change.

Can I run an A/B test with low traffic?

You can, but only for big changes, because small effects need a lot of users to detect. Test on the busiest step of your funnel and extend the test by a few full weeks if needed. If the calculated run time still goes past about 4 weeks, use user interviews or usability tests instead.

What is a good confidence level for an A/B test?

Common defaults are 95% confidence and 80% power, which keep the risk of a false winner low. Pick the level in Week 2, before the test starts, and keep it for the whole test. Look at the confidence interval too, because it shows how large the effect might really be.

Should I run several A/B tests at the same time?

You can run tests at the same time on separate pages or flows that users reach independently. Avoid two tests on the same page or step, because each one changes what the other measures. A roadmap with one test lane per page makes these overlaps easy to spot.

Generate this roadmap with AI