Questionnaire Sample Length: The Long and Short of Participant Engagement

Questionnaire Sample Length

Survey length in online interviews continues to be a bone of contention among people all along the marketing research continuum. While some within the industry recommend keeping questionnaires short to promote participant engagement, others often dismiss this advice. Short questionnaires, they assert, won’t unearth the depth and breadth of information that is needed. As a sample provider offering guidance based on research, SSI suggests keeping online interview length at 20 minutes or less. Generally, when interview length increases, fatigue also increases and, conversely, attention span decreases potentially damaging data integrity. In effect, researchers who insist on longer questionnaires, sincerely believing they’ll get more information, may, in actuality, be sabotaging their efforts. So its time for you to review your ongoing questionnaires.

Research Design

In order to more fully understand the possible effects of survey length, fatigue, and subsequent response quality, SSI recently fielded two surveys: one long and one short. This study replicated a ground-breaking study conducted in 2004 by Sandra Rathod and Andrea la Bruna which concluded, among other things, that data quality suffers as interview length increases.

The surveys, both then and now, utilized a block design and were divided into four blocks of questions, each representing a different subject matter. The blocks were randomized for each respondent so that the effect of survey length on response quality could be investigated by comparing whether the different order of the blocks led to different response patterns as the block position varied in the survey.

Fatigue Effects

One of the hypotheses of the 2004 study was that respondents would take less time and exert less effort later in the questionnaire than they would earlier in the questionnaire, due to fatigue. This hypothesis was proven in 2004: as the same block of questions was moved further back in the study, the time taken to complete it gradually reduced.

It could be argued that the decrease in block completion time was due to increased familiarity with the question set. It is true that the question blocks were similar in their construction and contained somewhat similar questions. However, additional evidence on panelist fatigue shows that at least some of the increased speed was due to fatigue.

Panelist Fatigue and Satisficing

One of the behavior outcomes of cognitive fatigue is satisficing – doing just enough work to satisfy the task. To see if this behavior was present in the 2004 study, researchers looked at a question, in each block, that it was possible to skip. This question offered a set of scales, presented in the form of sliders. The slider bar was positioned at the mid-point so it was possible to click on “next” without moving the slider and still leave some data behind. The likelihood of skipping the question rose as the skippable question was encountered further and further into the online questionnaire.

In 2009, SSI found precisely the same pattern. The first time the skippable question was encountered it was more likely to be completed than when it was seen on subsequent occasions.  This was particularly true for the long survey. A reduction in elapsed survey times did not mitigate the effect. The long survey, at nearly 25 minutes, was still too long.

Conclusions

In both 2004 and 2009, the long survey proved itself too long. It fatigued the respondent and led to satisficing behavior. When questions could legitimately be skipped, they were. Perhaps the most unsettling finding was that the instances of cheating, deliberately telling a falsehood in order to skip an entire section, also increased as the survey progressed.

Following the 2004 study, researchers indicated that there is a “critical point in online survey response when the fatigue effects become significantly more pronounced. That critical time is around the 20-minute mark. If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and responsive respondents.”

Today’s research confirms that interview lengths of 20 minutes or less can produce wonderful and engaged responses if well designed. The fact that there was much less satisficing and cheating in the short survey attests to this. If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and engaged participants.

About the Author: Pete Cape is Global Knowledge Director for Survey Sampling International. SSI provides access to more than 6 million research respondents in 72 countries. Sources include SSI proprietary panel communities in 27 countries and a portfolio of managed affiliates. SSI can potentially access anyone online to give their opinions via a network of relationships with websites, panels, communities and social media groups.