Readex Review

March 2010

Tips for Evaluating Survey Research Proposals

Make sure to compare apples-to-apples

Omissions that Raise Red Flags

Successful surveys start with proposals that outline essential survey details to confirm the shared expectations of the project.

Proposals that fail to address each of these issues call into question the level of research expertise that will be involved in the survey's execution:

  • Sample size
  • Expected response rate
  • Estimated number of responses for each segment of interest
  • Maximum Sampling Error (MSE)
  • Size and scope of questionnaire
  • The reports you'll receive
  • Survey schedule
  • Price

 

Evaluating survey research proposals can seem overwhelming, even to a well-versed researcher. Comparing proposals "apples-to-apples" is crucial to establishing which one will best meet your needs. Consider these ideas to help you focus on the details that contribute to a successful survey.

Make Sure the Proposal Responds to Your Objectives

The proposal process begins before the research firm offers you their take on how they recommend you conduct your survey and for what price. Instead, the process begins with the first discussion you have regarding the survey. Did the researcher take the opportunity to ask you specific questions about your objectives, the group of people you'd like to survey, and your ultimate goals? Details regarding your situation should pop up throughout the proposal. Surveys whose proposals do not incorporate your individual needs cannot possibly meet them.

For example, one research firm may recommend a very straightforward survey plan with a low price tag. Another may take time to recommend a more involved approach that considers your needs and objectives, but at a higher price.

Ask the higher priced firm to account for the differences. They should have no problem explaining the rationale behind their recommendations. On the other hand, have the low cost provider explain why they believe such a straightforward survey is right for you.

Sampling Plan

When reviewing the sampling plan, make sure the proposal mentions sample size, response rate, number of responses, and maximum sampling error. These figures help you determine whether you'll be able to confidently project the survey results to your entire population of interest. If you're unsure of the impact these figures have on the quality of your results, ask the researcher. They should be able to explain them in terms you can understand.

If you're interested in learning details about specific segments of your circulation, make sure that the sampling plan accounts for them. Sometimes a simple random selection of names is sufficient. In other cases small, but important groups should be over-sampled in order to collect enough responses to tell a story about the group.

Questionnaire

The quantity and types of information sought from respondents will impact cost. Quantity encompasses the number of questionnaire pages and number of variables to process. Type refers to how the questions will be processed, the data entry involved, and whether all, or just some data will be cleaned.

No evaluation is complete until you know the approximate number and types of questions planned for the survey. The number of open-ended questions should be included as well, because open-ended questions that capture verbatim responses can impact the response rate and the price of your survey. While these details can change during design, knowing the starting point helps establish what additional questions, pages, or transcribed questions will add to your bottom-line.

In addition, make sure the proposal clearly indicates who will develop the questionnaire content and whether it includes enough collaboration time to be sufficiently customized to meet your particular needs.

Data Collection Approach

For online surveys that invite respondents via email to respond to a web-based survey, paying attention to the data collection series can mean the difference between conducting a successful survey and one that frustrates your circulation. Multiple emails to respondents can encourage response by arriving at the inbox at a variety of times, some of which will be more convenient to your sample than others. However, follow-up emails should only be sent to non-respondents. Out of privacy concerns, sample members should be given the opportunity to opt-out with each contact. Outbound emails should be coded to only allow one response per person, and to prevent others from taking the survey.

Proposals for mailed surveys should clearly outline the data collection series and each component of the survey kit. A sophisticated mailing series can efficiently improve response rates and increase the quality of data. Some cost effective techniques that can boost response rates include the use of incentives, stamped reply envelopes, follow-up survey kits to non-respondents, alert letters or postcards, and personalization.

Data Processing

Your proposal should highlight the steps the research company will take to ensure that the data is accurate and representative. Depending on the type of survey, checking logic, consistency, and outliers can take a significant amount of time. There should be some process noted to identify inconsistent answers for surveys that collect a significant amount of numerical data (salary survey, market studies, budget planning). Finally, some percentage of mailed surveys need to be verified for data entry accuracy.

Analysis

Straightforward analysis of survey data can meet many objectives. In other cases, multivariate statistical analysis will provide deeper insights to achieve your objectives— making results easier to use. If your objectives include learning about separate segments of your circulation, crosstabulations should be specified.

Deliverables

A variety of reporting options exist for a survey: data tables, a summary of the results, in-depth analysis, graphed presentations, etc. As a result, you need to understand exactly what you'll receive following your survey and in what format.

If report and data table samples aren't offered, ask for them. You want to make sure that data tables are easy to read and attractive. Consider how well write-ups enhance the clarity and usability of the results. If you plan to use your reports in presentations, make sure they'll reflect well on you, or consider the time you'll have to take to reformat the results.

No Surprises!

Make sure the proposal covers all the bases: what you need to do and provide, what the firm will do, when they will do it, and how much it will cost.

There should be no surprises in what you need to supply. No "you need how much letterhead and envelopes?" a week before your survey is scheduled to mail.

Review the price carefully and understand what it includes and doesn't include. Avoid the "you want how much to report the results?" after your survey is complete.

Proposals with specific timelines will give you an idea of how long each step of the survey takes and illustrate the total time necessary to complete the project. If you have a deadline of when you need results, make sure it's addressed in the proposal. The annual meeting to go over survey results will be much less engaging if the reports won't be ready in time.

As with many things, you usually get what you pay for from a research provider. You'll almost always be able to find a firm that says they can do the same survey quicker and cheaper than another.

Beware of firms that cut corners by skimping on the time and energy it takes to do your survey right. That's the reason you're using a survey research firm in first place, right? Consider these tips as you evaluate proposals for your project. They should help you find the best value for your research dollar.

Back to Resources & Newsletter