How Response Rate Affects A Survey
Non-response bias is the error resulting from distinct differences between the people who responded to a survey versus the people who did not respond. Although this error’s impact on results cannot easily be quantified as sampling error can, response rate can be used as one way to gauge the potential for non-response bias. The higher the response rate of a survey, the lower the risk of non-response bias.
Look at the example below to see the effect that non-response bias might have on survey results. Let’s say you conduct three surveys of the same sample of ten students hoping to learn the percentage that received A’s in a recent class:
Only two of the ten students respond. They both received A’s, so your survey reports 100% of the students received A’s.
Four of the ten students respond, including the two who received A’s, so according to this survey 50% of the students received A’s.
Six of the ten students respond. With two receiving A’s, this survey reports that 33% of the students received A’s.
So, what impact did non-response bias have on each of our three surveys?
In reality three of the ten students (30%) received A’s.
What does it mean?
Here’s the breakdown of the potential magnitude of non-response bias for each of the three surveys:
Survey 1 = 70 Percentage Points Off
Survey 2 = 20 Percentage Points Off
Survey 3 = 3 Percentage Points Off
These surveys illustrate how non-response bias can affect how well data represents the population being surveyed. If someone relied on Survey 1 data for decision-making, they’d be way off. Survey 2 is much closer to the actual, but is still not a very good estimate. Survey 3, though, with a higher response rate, gives a pretty close estimate of the true value—only 3 points off.
Non-response bias can become an issue when there are distinct differences between the people who respond to a survey and the people who don’t respond. For example, students receiving A’s may feel more confident sharing their grade than the one who received an F—even if the survey is anonymous.
Keep this example in mind the next time you’re using survey results to assist in decision-making. Review the question. If there’s a reason why non-respondents would answer differently than those who did respond, consider the potential impact of non-response bias, and remember—the higher the response rate, the less likely error from non-response bias is to occur.