Survey Length vs. Data Quality

What does survey length have to do with data quality? This is a question being asked more frequently in market research circles. It is certainly a topic I have devoted much time to, for the simple reason that lengthy and complex surveys make it more difficult for all researchers to collect valid and reliable data. I touch on this further in another recent post (Keep it short and simple).

How long should a survey be? Surveys range from the quick poll with its one question to 30 – 40 minute marathon exercises which try everyone’s patience. It seems as if clients want respondents to give them 40 minutes of undivided attention, but therein lies a disconnect as most respondents are not motivated to do so.

What do we know about those who are willing to sit through these extended exercises? For starters, they tend to be quite different from your typical customer. They are heavy Internet users who tend to browse and buy online more often than the general population.

August 1 picWhat other evidence is there? Well, the Pew Research Center has reported their response rates dropped from 36% to nine percent between 1996 and 2012. ESOMAR in its 2014 Global Market Research Report showed a decline in the proportion of research dollars spent on surveys. There has also been a rise in satisficing behavior, or simply doing whatever it takes to complete the exercise in order to obtain the incentive. A 2011 study reported that as question number increased the amount of time spent per question decreased (see graphic below). Longer surveys lead to straight-lining, reduced number of selects from multiple response questions, and verbatim that have fewer words (if they are completed at all).

Then we have the rise of the mobile phone respondent. Surveys designed for the computer or tablet, do not translate well to the mobile phone, despite their increasing size. The experience has to be simplified in order for mobile users to respond.

What can we do to combat these trends? In this day and age of technology we can leverage it to our advantage in a few ways. First, we can employ more of a didactic approach where we ask a question and wait for the response. This works with community forums where ongoing discussions can be monitored. Mobile research is useful for brief, “in the moment” surveys where the experience of a purchase is fresh. Observational data from market experiments offers a quick read on the effects of changing stimuli.

I also advocate minimizing survey length by moving toward shorter surveys focused on specific topics. If you have developed a panel or an MROC (marketing research online community) it is more effective to leverage shorter, targeted surveys addressing detailed market concerns. If your surveys are regularly exceeding 20 minutes in length then you are not getting the quality of data necessary for decision making. Take a step back and look at ways you can increase the breadth (not depth) of your information collection system.

Image courtesy of Tuomas Lehtinen at

The author: Greg Timpany directs the research efforts for Global Knowledge in Cary, North Carolina, and runs Anova Market Research. You can follow him on Twitter @DataDudeGreg.


Speak Your Mind