We’re often asked if survey results are “projectable.” Although it masquerades like one, it really isn’t a yes or no question. Here you’ll learn how to use details about sample and size to decide if the result is projectable.
All Results are Projectable…
When 3 people answer a question and 2 say no, the result of 67% is projectable to that group of three people.
…The Real Question is “Who do the results represent and with what reliability?”
The biggest question when it comes to projectability is what group of people does the statistic represent? This is where sampling rears its head.
As a starting point, the results represent the group of people from which the sample was pulled—and no one else. Here are some examples:
Once you get an idea of who the results actually represent, you can get an idea of how well the results represent that group. That’s where response quantity and response rate come into play.
Response quantity directly relates to Maximum Sampling Error (MSE), a measurable form of error. It’s the +/- figure often quoted with statistics. So a 5 percentage point MSE at the 95% confidence level means that 95% of the time the actual average for the entire population will be within the range of 5 points lower and 5 points higher than the reported statistic.
So…if a result is 65% with MSE of +/- 5 percentage points at the 95% confidence level, then the true population value would be between 60% and 70% 95 percent of the time.
MSE is based on the quantity of responses: the more responses, the lower the MSE. Unfortunately the relationship is not linear. To cut MSE in half, you have to quadruple the number of responses.
If a statistic is based on 400 results, MSE hovers around +/- 5 percentage points. On the other hand, if a statistic is based on 100 results, MSE is closer to +/- 10 points. That can make a big difference in how you interpret a statistic.
Maximum Sampling Error is primarily based on the number of responses the survey yields: the more responses your results are based on, the lower the error. Unfortunately, as you can see from the graph below, the relationship is exponential rather than linear. For example, in order to cut the MSE in half, you need to quadruple the number of responses.
Another consideration regarding projectability is response rate, the percentage of people who were asked to participate in the survey that actually submitted a response. While this indicator of quality is not mathematical, the higher the response rate the better. Here’s why: consider how the people that responded were different from the people who didn’t. If you get a 95% response rate, the differences of the 5% of non-responders isn’t as critical as if you received a 10% response rate and had to wonder how 90% of those sampled differ from respondents.
A great illustration of this phenomenon can be taken from phone surveys. Let’s say that a phone survey of social behaviors is being taken on a Saturday night. Only people who are at home at that time will have the opportunity to participate. The answers from those who respond (people at home on a Saturday night) could be markedly different than the answers from those who were called to participate that didn’t (people out on a Saturday night).
Subject matter can also have a huge impact on how relevant response rate may be on results. Look at what is being asked. Could a low response rate indicate that you’re only hearing from respondents with a bone to pick or from those who wish to sing praises?
So, is this projectable? Consider this:
Use these three considerations to give you an idea of how “projectable the data is.” You can base judgment of the statistic on the following:
Who was asked to participate in the survey?
This is the group of people that the results represent.
How many responses is the statistic based on?
Through MSE, it tells you to what extent the statistic represents the group.
What percentage of people who were invited to take the survey responded?
You can determine whether it is likely that those who didn’t respond answered much differently than non-responders would have.
Remember: Survey Research is Based on Error
It’s important to remember that market research is based on mathematical theory, but there is error built into the system. Researchers are faced with the challenge of balancing budgets with their desire for exactness, yet no survey results offer laser beam precision. Instead, when you rely on measurement and judgment you can determine the statistical and practical significance of a statistic, and in turn, what that means to you.