Picture a study on the driving experience that surveys only motorcycle riders. The results say nothing useful about drivers. That’s sampling bias: a sampling error that happens when a study’s participants don’t represent the population it claims to describe.
In this blog, we’ll break down what sampling bias is. We’ll cover the six types that show up most often in research. We’ll also walk through real examples and the practical steps that keep a sample honest.
What is sampling bias?
Sampling bias is a systematic error. It occurs when some members of a target population have a higher or lower chance of being selected than others. It differs from ordinary sampling error, which is random and shrinks as a sample grows. Sampling bias does not shrink with a bigger sample. It just produces a more precise estimate of the wrong number.
The clearest example is election polling. A researcher who polls 1,000 middle-class, blue-collar voters gets a sample that skews toward one demographic slice. That sample ignores the income, education, and regional variation that shape how people actually vote.
Sampling bias, selection bias, and sampling error get used interchangeably. They describe different problems. The table below separates them, since mixing them up leads to the wrong fix.
| Term | What it actually means | Fixed by a bigger sample? |
|---|---|---|
| Sampling error | Random variation between a sample and the population, present in every study | Yes, it shrinks as sample size grows |
| Sampling bias | Systematic over- or under-representation caused by how participants were selected | No, a larger sample just repeats the same skew |
| Selection bias | The broader category; includes sampling bias plus bias introduced after selection, such as who drops out | No, needs a fix at the source, not more responses |
A representative sample needs a sampling frame that actually matches the population you’re trying to describe. When the frame is off, statistical cleanup can’t fully repair it.
What causes sampling bias?
Sampling bias comes from two places. One is how a study gets designed. The other is how it actually gets run in the field.
- Poor methodology. The sampling frame excludes or under-samples part of the population from the start. This often happens because researchers default to whoever is easiest to reach instead of defining the target population first.
- Poor execution. A sound design gets undermined when fieldworkers skip hard-to-reach respondents. It also breaks down when they stop following up with non-responders, or swap in convenience sampling instead of the random method the study called for.
Both causes point to the same failure point: the moment a researcher trades randomization for convenience.
What are the types of sampling bias?
Six patterns account for most sampling bias in market research and academic studies. Each one skews results in a specific, recognizable direction. Naming the pattern is the first step to correcting it.
Undercoverage bias
Undercoverage happens when a sampling method structurally cannot reach part of the population. It’s one of the most common drivers of sampling bias, and it often stays invisible until someone checks who got left out.
Online-only surveys are a frequent example. Only 14% of adults 65 and older say they’re online almost constantly, compared with 63% of adults 18 to 29. A web-only survey misses exactly that gap.
- People without reliable internet access
- People who distrust online surveys and decline the format
- Non-English speakers, when the survey is only fielded in English
Fixing undercoverage means adding an offline or mixed-mode channel. Simply growing the online sample won’t do it.
Voluntary response bias
Voluntary response bias is also called self-selection bias. It occurs when respondents who feel strongly about a topic opt in more often than everyone else. The people who show up aren’t neutral, so their answers overrepresent one side of an issue.
Call-in radio segments were the classic example. A more current one is any open social media poll. The people who click already have an opinion worth defending. The quietly indifferent majority just scrolls past. A survey with no invitation or quota controlling who gets a chance will almost always run into this.
Survivorship bias
Survivorship bias happens when a study only looks at the participants, companies, or products that made it through some selection process. It ignores the ones that didn’t.
- Startup research that studies only companies still operating today, skipping the majority that failed
- Product reviews that capture only customers who kept using the product, missing everyone who abandoned it early
- Investment performance studies that exclude funds that shut down mid-period
Each version produces results that look more successful than reality. The failures never entered the sample.
Non-response bias
Non-response bias shows up when the people who decline to participate differ systematically from the people who do. It isn’t the same as a low response rate. A survey can have a low response rate without much bias, as long as the people who skip it resemble the people who answer.
The bigger risk is when non-response correlates with the topic itself. Sensitive questions about income, health, or personal habits push away exactly the respondents whose answers would matter most. Telephone survey response rates show the scale of the problem: Pew Research Center’s typical phone survey response rate fell from 36% in 1997 to just 6% by 2018.
Recall bias
Recall bias occurs when respondents can’t accurately remember the information a study asks about. It’s one of the few sampling biases that has little to do with who was selected.
Memory fades and reconstructs itself over time. Answers about past behavior drift toward round numbers and recent events. Two adjustments help:
- Interview respondents close to when the experience happened
- Use diaries or records instead of relying on memory alone
Neither fully removes recall bias, but both shrink it considerably.
Observer bias
Observer bias comes from the researcher, not the respondent. It happens when a researcher’s expectations shape how questions get asked. It also shows up in which responses get emphasized or which data points get double-checked.
This is usually subconscious rather than deliberate. A researcher who expects a certain result may phrase a follow-up question differently for participants who confirm it. Structured question wording helps. So does blinded coding of open-ended responses and a second reviewer for qualitative analysis.
What are some real-world examples of sampling bias?
The patterns above show up constantly once you know what to look for. Here are a few, both classic and current.
- Election polling.
Surveying 1,000 middle-class, blue-collar voters and treating it as representative of an entire electorate ignores the income, education, and regional variation that predict how people vote.
- A city traffic law study.
Interviewing shoppers outside one mall misses residents without transportation to that mall, people who avoid malls, and anyone who shops elsewhere. That’s an undercoverage problem hiding inside a convenience sample.
- AI and machine learning models.
A model trained mostly on one demographic’s data can perform worse for everyone else. IBM’s research team lists this as one of the clearest modern examples of sampling bias in practice.
- Product satisfaction research.
Surveying only current, active customers and skipping people who churned produces a survivorship-biased picture of satisfaction.
How does sampling bias affect research results?
Sampling bias narrows what a study can honestly claim. Findings from a biased sample only generalize to people who resemble that sample, not to the full population the research was meant to describe.
This matters most when the bias correlates with the outcome being measured. A customer satisfaction score pulled only from people who bothered to respond usually runs higher than reality, since satisfied customers are more willing to spend two minutes on a survey. Decisions made on that inflated number, from staffing to product roadmaps, inherit the same distortion. The danger isn’t that the numbers look obviously wrong. It’s that they look confident and precise while quietly describing the wrong group of people.
What common mistakes lead to sampling bias?
Most sampling bias traces back to a handful of avoidable shortcuts, not to bad luck.
- Treating a large sample size as proof of representativeness, when size doesn’t fix a skewed selection method
- Defaulting to convenience sampling because it’s faster, without checking who that method leaves out
- Closing the field period without following up on non-respondents at least once
- Reusing an old sampling frame or mailing list that no longer matches the current population
- Letting an opt-in panel or online community stand in for the general public without weighting it
How can you avoid sampling bias?
You can’t eliminate sampling bias entirely. But you can control it with a few deliberate design choices made before fieldwork starts.
Start by defining the target population and sampling frame together. Then check that the two actually match. From there:
- Default to probability sampling methods over convenience sampling whenever budget allows
- Keep surveys short enough that length itself doesn’t push out busy or less-motivated respondents
- Make the survey accessible across channels (online, phone, in person) instead of one mode only
- Follow up with non-respondents at least once before closing the field
- Give every eligible person in the population a genuine, equal chance to participate
This takes more planning than a shortcut sample. It pairs well with the broader practice of avoiding survey bias at every stage of a study, not just sampling.
Why does stratified random sampling reduce bias?
Stratified random sampling reduces bias by forcing the sample to mirror the population’s makeup on the traits that matter most. It doesn’t leave that outcome to chance.
Steps to build one:
- Define the population and the variable to stratify by, such as gender, age, or region
- Divide the population into strata that don’t overlap and cover everyone
- Calculate each stratum’s share of the total population
- Randomly select respondents from each stratum in proportion to that share
- Combine the strata into one sample and confirm the proportions match
A population of 5,000 people split 50/50 by gender needs a sample of 100 built the same way. That’s 50 men and 50 women, not whatever ratio convenience sampling happens to produce. Learn more in this guide to stratified random sampling.
How do you measure or check for sampling bias in your data?
Checking for sampling bias means comparing your sample’s makeup against known facts about the population. Don’t just trust the response count.
| What to check | Compare against | What it tells you |
|---|---|---|
| Demographic mix (age, gender, region, income) | Census or industry benchmark data for the same population | Whether any group is over- or under-represented |
| Response rate by outreach channel | Response rate across other channels in the same study | Whether one channel is silently skewing who gets included |
| Early vs. late responders | Answers from each group on key questions | Whether late responders, a proxy for reluctant ones, differ meaningfully |
| Sample size vs. margin of error | Your target confidence level and margin of error | Whether the sample is large enough to detect real differences, using a sample size calculator |
If any row shows a meaningful gap, weighting the data usually helps more than collecting more of the same responses. Extending fieldwork toward underrepresented groups works too.
How do you choose the right sampling method to avoid bias?
The right method depends on the study’s goal. Does it need to generalize to a full population, or explore a narrower question in depth?
- Choose probability sampling (simple random, stratified, or systematic) when the goal is to generalize findings to an entire population, such as market sizing or election research.
- Choose non-probability sampling only for exploratory or qualitative work, such as early concept testing, where speed and depth matter more than representativeness. This overview of survey sampling walks through where each approach fits.
- Avoid defaulting to whichever method is fastest to field. The research question should set the choice, not convenience.
How does QuestionPro Audience help reduce sampling bias?
A sampling frame is only as unbiased as the pool of respondents behind it. Panel quality is part of the fix.
QuestionPro Audience gives researchers access to millions of double opted-in respondents. Each one is profiled on more than 300 data points, which makes it realistic to build a proportionate, stratified sample instead of settling for whoever happens to be reachable. Targeting by age, income, region, or dozens of other traits lets a researcher fill in the exact strata a study calls for. That beats discovering the gaps after the data is already in.
Why sampling bias deserves attention before fieldwork begins
Sampling bias is easiest to prevent at the design stage and hardest to fix afterward. That’s why it deserves attention before a single response comes in.
The organizations that get this right treat sampling as a deliberate decision. It isn’t a byproduct of whoever was easiest to reach. Three things are worth remembering:
- A bigger sample never fixes a biased one
- The type of bias usually points directly to the fix
- Checking the sample against real population data catches problems weighting alone can’t
Frequently Asked Questions (FAQs)
Sampling error is random variation that shrinks as a sample grows, present in every study. Sampling bias is systematic. It’s caused by how participants were selected, and it doesn’t improve with a larger sample, only a corrected selection method.
No. A larger sample only gives a more precise estimate of a skewed result if the selection method itself is biased. Fixing sampling bias means changing who gets included, not just collecting more responses from the same flawed frame.
Researchers commonly group sampling bias into six recurring types: undercoverage, voluntary response, survivorship, non-response, recall, and observer bias. Some fields add narrower variants, but most real-world cases fall into one of these six patterns.
Online-only surveys, opt-in panels, and convenience sampling carry the highest risk. Each one relies on whoever is easiest to reach, rather than a random or stratified draw from the full target population.
Partially. Statistical weighting can correct known demographic gaps after the fact. It can’t recover information from people who were never reachable in the first place. Prevention during sample design beats any after-the-fact correction.
Yes. Models trained on data that underrepresents certain groups tend to perform worse for those groups once deployed. It’s the same underlying mechanism as sampling bias in survey research, just applied to training data instead of respondents.
