Survey Bias

Survey Bias definition:

Survey Bias or response bias is a general term for a range of cognitive biases that influences participants to reply an inaccurate or dishonest response.

Want to learn more?

Download the entire glossary list in a printable list

Yes, I want this list

Types of bias:

- Researcher Bias

The viewpoint of the researcher has a way of creeping into the survey. All research designers are human and have their own views and opinions. Even the most practiced and professional researchers can have subtle biases in the way they word questions or interpret results.

- Poor match of the sample to the population

It is almost never the case that the sampling frame you use is a perfect match to the population you are trying to understand, so this error is present in most studies. You can sometimes recover from asking the wrong questions, but you can never recover from asking them of the wrong people.

Most people like to focus on questionnaire development when a new project is awarded. The reality is the sampling and weighting plan is every bit as consequential to the success of the project, and rarely gets the attention it deserves. We can tell when we have a client that really knows what they are doing if they begin the project by focusing on sampling issues and not jumping to questionnaire design.

- Lack of randomness/response bias

Many surveys proceed without random samples. In fact, it is rare that a survey being done today can accurately claim to be using a random sample. Remember those statistics courses you took in college and graduate school? The one thing they have in common is pretty much everything they taught you statistically is only relevant if you have a random sample. And, odds are that you don’t.

A big source of “non-randomness” in a sample is survey bias. 10% is considered a good response rate from an online panel. When we report results of these studies, we are assuming that the vast majority of people who didn’t respond would have responded in the same way as those who did. Often, this is a reasonable assumption. But, sometimes it is not.

- Failure to quota sample or weight data

Even if we sample randomly, it is typical for some subgroups to be more willing to cooperate than others. For example, females are typically less likely to refuse a survey invitation than males, and minorities are less likely to participate. So, a good researcher will quota sample and weight data to compensate for this. In short, if you know something about your population before you survey them, you should use this knowledge to your advantage.

- Social desirability bias

This is a form of survey bias in responses caused by respondents' desire, either conscious or unconscious, to gain prestige or appear in a different social role.