They say to err is human, and this can certainly apply to the surveys we create. A good survey can yield a wealth of valuable data, however, when there are sampling errors that data can be skewed. When the time comes to analyze that flawed data, the result can lead to bad decision making and incorrect estimates or inferences made about the population. So, why do sampling errors occur? In this blog, we’ll look at the most common types of issues and how to reduce sampling errors.
Create your FREE Survey, Poll, or Questionnaire now!
What is a Survey Sampling Error?
A survey sampling error occurs when a sample – a subset of a larger population – differs significantly from that greater population. While there is almost always going to be some amount of sampling error in any survey (which is why most surveys highlight a margin of error, usually between 4% and 8% at a 95% confidence level), sampling errors can make this margin unacceptable for drawing any type of conclusion. Read more about sampling margin or error and how to calculate margin of error.
Sampling Errors vs Non-Sampling Errors
It’s important to understand that sampling errors are different from non-sampling errors. Non-sampling errors aren’t based on the participants chosen to partake in a survey, but rather the survey design, such as the order of questions, or the survey questions themselves (to write survey questions like a pro, get our free ebook here). There are a variety of questions that, whether intentionally or not, introduce non-sampling errors into a survey, such as:
- Leading questions which include wording that sways participants one way or another or provide more options on one side of a scale question.
- Loaded questions which force respondents to answer a question in a particular way by not providing answers such as “not applicable” or “other.”
- Double-barreled questions which put two questions into one, leaving respondents unsure of how to answer.
Non-sampling errors also cover issues like response bias, where the number of people who answer the survey are disproportionately represented. For example, let’s say you survey retired and non-retired people. Your sample may be balanced but the result may not be. Why? Because retirees with ample free time compared to those who work may be more inclined to respond.
7 Types of Survey Sampling Errors & How to Avoid Them
Here’s a look at the seven reasons for sampling errors including several survey sampling examples.
1. Sample Frame Error
What it is: This occurs when a researcher targets the population subset incorrectly. For example, selecting a sampling frame from the phonebook, which would eliminate anyone who is unlisted or doesn’t have a landline. These types of errors are known as “erroneous exclusions,” and is what happened when the Chicago Daily Tribune published the headline “Dewey Defeats Truman” in 1948. What went wrong? The paper had conducted surveys by phone to predict a winner, but only the wealthy – who favored Dewey – had phones back then. Lower- and middle-class families – who favored ultimate winner Truman – were left out leading to the infamously incorrect headline.
How to avoid it: Consider increasing the population size and ensuring that most of the selected respondents adequately represent the rest of the population. Be sure that everybody has the opportunity to participate as well. While many surveys are no longer conducted by telephone but rather email, sample frame errors can still occur (for example, the elderly may not use email, so if you’re surveying them you may need to consider multiple types of survey methods).
2. Selection Error
What it is: This type of sampling error is the result of only volunteers opting into a study. It is likely that these people feel strongly about the topic and want their voice to be heard. However, this leaves out people who don’t feel as strongly, and their opinion also needs to be included.
How to avoid it: Request responses from non-volunteers. Follow-up with emails or other survey methods. You may also consider offering a survey incentive to get them to participate in order to obtain a well-rounded and balanced sample.
3. Non-Response Error
What it is: When you don’t receive responses from every unit in the sample group you will be faced with a non-response error. These errors result in a smaller sample size leading to a larger margin or error. Survey bias can also be introduced in this scenario as the non-respondents may have different opinions from those who did respond.
How to avoid it: Be sure non-respondents are able to use the medium in which the survey was conducted (again, this may mean employing several types of survey methods). If you’re surveying a highly diverse group, be sure the survey is translated into multiple languages (you can also use surveys with images, which help cross language barriers). Finally, try following up or consider offering an incentive if they simply aren’t interested but their opinion could be valuable.
4. Population-Specific Error
What it is: This is a common type of sample error when a researcher is unsure of exactly who to target. For example, a survey about health issues among the elderly. Who should be surveyed? The elderly people with health issues, their caregivers, or their physicians? Another example would be back to school needs. Should the researcher survey students, their parents, or teachers?
How to avoid it: In survey planning stages, be sure to be confident on what you’re hoping to understand and who would be best able to provide the most valuable input. Let’s consider the examples above:
- Let’s say you’re surveying about a new Alzheimer’s drug. Although the elderly are the ones experiencing the problem, the caregivers or physicians will probably be the ones to determine if it’s the right path, so their opinion is most important.
- For a survey about back to school needs, such as apparel, kids may influence purchase decisions. However, it’s ultimately the parents making the purchase so they would be the most appropriate target.
Another option, obviously, is to survey all those who may be involved. Of course, this can be time-consuming and expensive, so be sure to mind your budget!
5. Undercoverage Error
What it is: This common error simply means there is a disparity in the representativeness of respondents. This is most likely to occur when the researcher doesn’t plan the sample carefully or uses a type of survey that limits who can respond to it.
How to avoid it: Researchers can help eliminate representation errors by creating a well thought out sample design, using a large enough sample to reflect the entire population. A website or email survey is a great way to avoid representation errors because there are no geographical limitations and almost all age groups are online (even within the 65+ age range, 75% are online).
6. Convenience Sampling Error
What it is: Convenience sampling is a popular way to get quick, easy, and inexpensive results. By definition, convenience sampling is when the researcher surveys those who he or she has easy access to, without regard to the larger population. For example, a researcher studying workplace happiness who only surveys people at the company located next door. It’s possible that the majority of people at that company are unhappy. Of course, that doesn’t necessarily represent the majority of people at any other company. The researcher, then arriving at the conclusion that “most workers are unhappy at their job,” could potentially be inaccurate.
How to avoid it: Avoid using the convenience sampling method for anything other than non-scientific or “fun” surveys. Or, use it as a starting point to get some quick impressions and then employ a probability sampling method, such as simple random sampling, which helps avoid bias.
7. Researcher Bias
What it is: Some samples are chosen deliberately by a researcher rather than randomly. For example, a store owner may stand outside his store and survey shoppers about their experience as they leave. In theory, this should be random, but the researcher may have personal bias. He may stop only people who are of a certain gender, age, race, ethnicity, and so on. This leaves out a wide swath of other demographics who may have differing opinions.
How to avoid it: Reduce sampling errors in research by using more randomized sampling methods, e.g. stop every fifth shopper regardless of who they are. Another way to avoid researcher bias is to install a survey kiosk at the store; a computer does not care what demographic a shopper falls into, and the survey will be open to everyone.
Conclusion
Surveys are a valuable tool for researchers and marketers. However, when sampling errors are introduced they can yield a study worthless at best, and dangerous at worst. Reacting to incorrect data can even sink a study or a company! So, be sure to avoid the seven types of sampling errors we highlighted here.
Still concerned about sampling errors? Try using SurveyLegend for your surveys, questionnaires, polls, and forms. You can send your SurveyLegend surveys online, helping you reach large sample groups and response rates. Surveys can be quickly and easily designed, and you can include survey pictures too. By adding images into your survey, you can be sure your Q&As are easily understood by all participants. Our surveys can also be taken via survey kiosk, further helping eliminate researcher bias. It’s free to start, so sign up today!
Are you concerned about sampling errors in your surveys? What do you do to reduce sampling errors? Let us know in the comments.
Create your FREE Survey, Poll, or Questionnaire now!
Frequently Asked Questions (FAQs)
A survey sampling error occurs when a sample, which is a subset of a larger population, differs significantly from that greater population.
A non-sampling error has the proper sample of participants, but errors are introduced as a result of poor survey design or poorly worded questions.
There are seven types of sampling errors:
– Sample Frame Errors
– Selection Errors
– Non-Response Errors
– Population-Specific Errors
– Undercoverage Errors
– Convenience Sampling Errors
– Researcher Bias
