Cure Conflicting Survey Results with Representative Survey Methods

What would you do if you…
...a survey found 75% of customers used your product daily, but the figure was actually 61%?

In each of the scenarios to the left, both sets of percentages are actual results from a single survey. Some of the percentages came from respondents for whom email addresses were available. They were surveyed via an email/online method. The other percentages came from those who were surveyed via traditional mail survey because no email addresses were available for them.

To be Accurate, Surveys Must be Representative

The results illustrate the sometimes dramatic differences between two groups of respondents in the same population, and probably raise more questions than they answer. What exactly happened in that survey, and how could we possibly trust survey data after reading those results?

Well, surveys conducted against a small but representative group can accurately portray the needs, interests, and behaviors of the larger group from which they are selected; however, surveys must be executed correctly in order to yield reliable results.

What is easy to gloss over is the word “representative.” A sample that is selected from the entire population represents the entire population. One that is chosen only from those who have provided email addresses does not.

When a land address is typically available for all those in the population, the simplest way to overcome this representation challenge is to execute a mail survey. The sample will be selected from the entire population, so the responses gathered will accurately represent the entire population. Furthermore, mail surveys typically achieve a relatively high rate of response which increases representation even further.

While a mail survey may be a straightforward way to gather representative results, unfortunately, when research budgets get cut, traditional mail surveys may not survive the chopping block. On the other hand, as we have just seen, an email/online approach, while inexpensive, may not yield representative results … and will cost you time and money.

What to do?

The answer is to capture data using a combination of an email/online survey and a traditional mail approach. This mixed mode solution capitalizes on the cost-effectiveness of the email/online method while accounting for the group of people who haven’t provided an accurate email address. As illustrated in the examples above, leaving out this group of respondents would likely bias the results.

To balance budget and representation, two samples are selected from the population. One sample is selected from those for whom email addresses are available. These individuals are contacted by email, and are directed to a web-based survey. The second sample is chosen from those who cannot be contacted by email (and can include those whose emails were undeliverable, as well as those who did not respond). This group is contacted and will respond via traditional mail survey.

Once the responses are gathered, the resulting data is weighted to reflect actual population proportions. Weighting avoids over-representation of one group over the other and is required to yield accurate and unbiased data from a mixed-mode survey.

Where to Use the Mixed Mode Technique

The mixed mode method is ideal for any survey in which there are barriers to contacting all members of the population in a single way. When conducting surveys for decision-making or for persuading, though, the need for accurate representation is essential to meeting the survey’s goals.

During survey planning, consider your goals and the level of risk involved in trusting flawed data. In particular situations, a mixed mode approach offers peace of mind over less representative methods, and maximizes the research investment.