A/B Testing: What It Is, How It Works, and How to Run One

A/B testing is a controlled experiment that shows two versions of a page, email, or app to separate groups of real users and measures which one performs better against a set goal. It replaces opinion with evidence, so teams stop debating headlines and start reading results.

Done well, A/B testing turns small changes into steady gains. Done badly, it produces confident answers that are simply wrong.

In this blog, we will cover how A/B testing works, the main test types, a step-by-step process, sample size rules, real examples, and the mistakes that quietly ruin results.

What is A/B testing?

A/B testing, also called split testing or bucket testing, is a randomized experiment that compares a control (version A) against a variant (version B) to see which one moves a predefined metric more. The metric might be clicks, sign-ups, purchases, or survey responses.

Every valid test has four parts:

• Control (version A): The current experience that real users already see.
• Variant (version B): The changed version you want to test against the control.
• Random assignment: Chance, not the team, decides which group sees which version.
• Success metric: One measurable goal, set before launch, that decides the winner.

Random assignment is what makes the comparison fair. Traffic source, device, and time of day are spread evenly across both groups, so any gap in results traces back to the change itself.

How does A/B testing work?

An A/B test works by splitting live traffic, tracking one metric, and checking whether the gap between versions is bigger than chance would explain. Four moves make up the loop:

1. Split: The testing tool assigns each visitor to version A or B at random, often 50/50.
   
2. Show: Each visitor sees only one version for the whole test, so experiences never mix.
   
3. Track: The tool records the success metric for every visitor in both groups.
   
4. Compare: Analysts check whether the gap is statistically significant, meaning a difference this large would be unlikely if both versions performed the same.

What are the types of A/B testing?

Businesses run four main kinds of tests, and each fits a different change size and traffic level. Simple tests still dominate.

Classic A/B testing

Classic A/B testing compares two versions of one page where only one or two elements change, such as a headline or a button. Everything else stays identical, so the result points to that change.

• Best for: Focused questions about a single element.
• Trade-off: It answers one question at a time.

Split URL testing

Split URL testing, also called redirect testing, sends traffic to two separate URLs that each host a different page design. It suits full redesigns that are too different to build as a variant of the same page.

• Best for: Redesigns and new landing pages.
• Trade-off: You build, maintain, and quality-check two full pages.

A/B/n testing

An A/B/n test compares one control against two or more variants at once, such as four different headlines. It tests more ideas in a single run.

• Best for: Several competing ideas that share one hypothesis.
• Trade-off: Traffic divides across more groups, so each group fills more slowly.

Multivariate testing

Multivariate testing changes several elements at once and measures every combination. Three headlines and two images create six versions, for example. It shows how elements interact, not just which one wins alone.

• Best for: High-traffic pages that need polish after the layout is settled.
• Trade-off: It needs far more traffic, because combinations multiply with each added element.

How is A/B testing different from similar methods?

Several methods get confused with A/B testing because they also compare options. The table shows what each one compares and when to pick it.

People often use A/B testing and split testing as synonyms. Some teams reserve split testing for the separate-URL version. Treat both terms as the same method unless a vendor defines them differently.

What are the benefits of A/B testing?

A/B testing lets teams ship changes based on measured user behavior instead of internal opinion. Five benefits stand out:

1. Evidence over opinion: A test settles debates with data, so the loudest voice stops winning by default.
   
2. Lower risk: A change reaches a slice of traffic first, which limits the damage if it backfires.
   
3. Higher conversion rates: Tested wording, layout, and form changes lift the share of visitors who complete a goal.
   
4. Better return on existing traffic: Visitors cost money to acquire, so converting more of them beats buying more.
   
5. Compounding learning: Each result, win or lose, sharpens the next hypothesis.

A single test rarely transforms a business. A steady testing habit does.

Step-by-step guide to running an A/B test

A reliable A/B test follows seven steps, and skipping any one of them weakens the result.

1. Research the problem.
   Use analytics, heat maps (visual maps of where visitors click), session recordings, and surveys to find pages with high traffic and weak conversion.
   
2. Write a hypothesis.
   State what you will change, what you expect, and why. For example: “Adding ‘No credit card required’ beside the sign-up button will raise trial starts because visitors fear surprise charges.”
   
3. Choose one primary metric.
   Pick the single goal that decides the winner. Add guardrail metrics, which are secondary measures such as cancellation rate that must not get worse.
   
4. Build the variant.
   Create it in your A/B testing software or in code, and change only what the hypothesis names.
   
5. Calculate sample size and duration.
   Fix both before launch, using the rules in the sample size section below.
   
6. QA and launch.
   Check both versions across browsers and devices. Confirm tracking fires for each group.
   
7. Analyze, decide, and document.
   Read results against the plan you set. Then log the hypothesis, dates, numbers, decision, and lesson.

Client-side vs server-side testing

Testing tools apply changes in one of two places, and the choice decides who can build the test.

• Client-side: JavaScript in the visitor’s browser changes the page. Marketers launch faster with visual editors, but visitors may see a brief flash of the original page.
• Server-side: The server sends the correct version before the page loads. It needs engineering time, but it can test pricing, algorithms, and product logic that a browser cannot change reliably.

A worked example

Picture a SaaS company whose pricing page has a high bounce rate. The team hypothesizes that defaulting the billing toggle to annual will raise trial starts, because it shows the lower monthly price first. Trial starts are the primary metric. First-month cancellation rate is the guardrail.

They split traffic evenly for three weeks and reach the planned sample size. The annual-default version produces 9 percent more trial starts, and the result clears the significance threshold. This is a hypothetical example, so treat the 9 percent as an illustration, not a benchmark.

How do you decide what to test first?

Test ideas outnumber testing capacity, so prioritization decides whether a program learns fast or burns traffic. Start with evidence, not hunches. Surveys, heat maps, and session recordings show where visitors stall, and many website optimization tools bundle those methods.

High-impact tests to prioritize:

• Value proposition and headline
• Call-to-action wording and placement
• Checkout flow and form length
• Page layout and content hierarchy
• Pricing and offer presentation

Lower-impact tests to deprioritize:

• Button color on its own
• Minor font changes
• Spacing tweaks with no hypothesis behind them

Color rarely changes a decision. An unaddressed objection does. A sign-up page that adds “No credit card required” next to the button tackles a real worry, while a blue-to-green swap rarely does.

Score ideas with a simple framework

ICE scoring rates each idea from 1 to 5 on impact, confidence, and ease, then averages the three. Higher scores run first. The scores below are illustrative.

CXL’s PXL model, a scoring sheet, goes further. It swaps gut-feel scores for yes-or-no evidence questions, which reduces subjectivity.

When should you skip A/B testing?

A/B testing is the wrong tool when traffic is too low, the change is too risky, or no one can name a hypothesis. Skipping a test is a valid decision.

• Low traffic: A page with a few hundred visitors a month can take many weeks to fill a test. Test bigger changes, use higher-volume metrics such as clicks instead of purchases, or switch to interviews and surveys.
• No clear hypothesis: Random tweaks produce random results.
• Unmeasurable outcome: If you cannot track the goal, no test can score it.
• Compliance or safety risk: Some changes need legal or safety review, not a traffic vote.

Teams in these situations can still learn from users. Qualitative research often finds the problem faster than a test could.

How big should your sample be, and how long should a test run?

Sample size is the number of visitors each version needs before a test can detect a real difference reliably, and you should fix it before launch. Four inputs set the number:

• Baseline conversion rate: Today’s rate for the page you are testing.
• Minimum detectable effect (MDE): The smallest lift worth detecting, such as a 20 percent relative lift, which turns 10 percent into 12 percent.
• Statistical power: The chance of catching a real effect. Eighty percent is the usual standard.
• Significance level: The false-positive threshold, commonly 5 percent, which matches 95 percent confidence.

Here is a real calculation. A 10 percent baseline and a 20 percent uplift call for about 2,863 visitors per group. Smaller lifts need far more, and calculators differ in their assumptions, so totals vary. Try your own numbers, or read how sample size determination works.

How long should an A/B test run?

Run the test until it reaches the planned sample size, and cover at least two full business cycles. Cycles matter because behavior shifts with:

• Day of the week
• Traffic source mix
• Email and content publishing schedules
• Returning visitors who buy days later
• Paydays and other outside events

Skip magic numbers such as “100 conversions.” They ignore your baseline, effect size, and traffic pattern.

Do not stop early because the dashboard shows a winner. Results often swing in the first days and then drift toward no difference, a pattern called regression to the mean. In one e-commerce example, the early leader ended with no difference after four weeks. The novelty effect adds another trap, where a change looks strong only because it is new.

How do you read A/B test results?

Reading results means checking data quality first, then the primary metric, the size of the effect, and the guardrails. A winner on a broken test is still a loser. For the underlying math, see this guide to statistical analysis.

The five checks, in order

1. Data quality.
   Confirm the traffic split matches the plan. A mismatch, called sample ratio mismatch (SRM), signals a bug and invalidates the test.
   
2. Primary metric.
   Check whether the metric you chose moved in the planned direction.
   
3. Effect size and uncertainty.
   Read the confidence interval, which is the range of plausible true effects. Do not rely on the p-value alone. The p-value is the chance of seeing this result, or a more extreme one, if there were no real difference.
   
4. Guardrail metrics.
   Confirm nothing important got worse.
   
5. Planned segments.
   Check device or new versus returning visitors only if you set those segments before launch. Hunting through segments afterward finds false winners.

Statistical significance versus practical significance

A result can be statistically significant and still not worth shipping. A 0.1 percent lift may clear the threshold on huge traffic yet never repay the engineering cost. Decide the smallest lift worth shipping before the test starts. Then compare the confidence interval to that number.

Bayesian or frequentist statistics?

Testing platforms use one of two statistical approaches, and each reports results differently.

• Frequentist: Tests the no-difference hypothesis and reports p-values. It works best with a fixed sample size and no peeking.
• Bayesian: Reports the probability that a variant beats the control. Many stakeholders find it easier to read.

Follow your platform’s method. Do not mix one method’s results with the other’s stopping rules.

What do real A/B testing examples look like?

Examples make the method concrete, and they show how small changes can carry large stakes. Three cases show the range, from a famous win to everyday tests.

Bing: A headline change worth $100 million

In 2012, a Microsoft engineer tested a small change to how Bing displayed ad headlines. Managers had ranked the idea as low priority, and it sat on the shelf for more than six months, according to Harvard Business Review.

• Result: Revenue rose 12 percent, which would exceed $100 million a year in the United States alone.
• Lesson: Teams cannot rank ideas by gut feel, because cheap changes sometimes pay off enormously.

Checkout button text

An online retailer tests “Complete Order” against “Get My Order.” This is a hypothetical example.

• Primary metric: Completed purchases.
• Guardrail metric: Return rate, to catch wording that oversells.
• Lesson: Even tiny copy changes need a metric, a sample size, and a fixed end date.

Email subject lines and US compliance

Email is the easiest place to start testing, because the send is already split by design. In the United States, both subject lines must accurately reflect the message content under the CAN-SPAM Act. The FTC says each violating email can draw penalties of up to $53,088.

• Test ideas: Length, personalization, and question versus statement.
• Primary metric: Click-through rate.

What are the most common A/B testing mistakes?

Most failed tests fail on process, not creativity. Setup quality matters. Analysis found a 58 percent win rate for well-designed experiments against 35 percent for poorly configured ones. Those figures come from real customers, so treat them as directional. Eight mistakes show up again and again:

1. Stopping early: Peeking and calling a winner inflates false positives.
   
2. Changing several things at once: You cannot tell which change caused the result.
   
3. Skipping the sample size calculation: Underpowered tests miss real effects.
   
4. Ignoring tracking bugs: Run an A/A test first, because two identical versions should show no difference.
   
5. Testing trivial changes: Tiny tweaks rarely produce detectable lifts.
   
6. Fishing in segments: Slicing results after the fact manufactures winners from noise.
   
7. Letting tests collide: Overlapping tests on the same page or metric can contaminate each other.
   
8. Not archiving results: Teams repeat old tests when nobody logs hypotheses, dates, metrics, and decisions.

How do you run survey A/B tests in QuestionPro?

Survey A/B testing randomly assigns respondents to different versions of a concept, message, or question, then compares their answers. A/B testing in QuestionPro surveys uses the Block Randomizer, which is available with the Research Edition license. Businesses using QuestionPro Market Research Software can set these experiments up inside the survey builder.

What can you test with a survey experiment?

Survey experiments measure what people prefer and why, not what they click. Common tests include:

• Ad or message concepts before launch
• Product names, packaging, or price messages
• Question wording and answer order

Each respondent sees one version only. Researchers call this monadic testing, meaning each person evaluates a single concept so comparisons stay unbiased.

How to set up a block randomizer

1. Open your survey and click Tools.
2. Choose Block Randomizer, then click Add Randomizer.
3. Drag the survey blocks you want to compare into the randomizer section.
4. Select how many blocks each respondent sees. To show one version per person, select one block.
5. Save and launch. Later, export your data with the Include Randomization Data toggle on, so you can see which block each respondent saw.

Survey tests do not replace behavioral tests on a live site. A web experiment shows what visitors do. A survey experiment shows what they prefer and why. Many teams run both.

Why every A/B test is worth running, even a losing one

A/B testing works because it makes ignorance cheap to fix. Even at Google and Bing, only 10 to 20 percent of experiments produce a positive result. Businesses that expect a win every time quit too soon.

Treat each test as a question you are paying to answer. A losing test stops you from shipping a bad idea. A flat result shows the change does not matter, so you can spend effort elsewhere. Either way, you learn something no meeting could have told you.

Start small. Pick one hypothesis, one metric, and a sample size calculated up front. Then write down what happened. The habit, not any single test, is what compounds.

Frequently Asked Questions (FAQs)