All about the professional respondents
There is a paradox in the market research industry around respondents. We do all we can think of to convince people to take our surveys. When they respond enthusiastically to our pleas and become frequent responders, however, we tend to deride them as “professional respondents,” accuse them of “gaming the system” and doubt the accuracy of their answers. This contradiction can hurt our credibility with the public we are seeking to engage in our research.
Moreover, our distrust of frequent responders has been proven to be unfounded. Studies that Survey Sampling International (SSI) and our clients have conducted examining responses of frequent vs. less frequent responders show little if any, the correlation between response frequency and quality.
Frequent responders are not speeders
For instance, our research reveals no evidence that frequent responders try to speed through surveys. They are no more likely than less frequent responders, for example, to select “none of the above” on questions such as what brands they have purchased in a category over the last three months. In fact, they make a relatively high number of marks on “check all that apply” questions.
To illustrate that point, consider a section of one survey that presented a “brand imagery” grid question that asked respondents to indicate which of the 17 – 19 statements (depending on the category) exemplified each brand of which they were aware. The frequent responder group indicated, on average, that 46% of the statements applied to a brand included in three categories. Among the entire group, respondents, on average, associated only 40% of the statements with a brand.
A second survey, which focused specifically on the effects of panel tenure and response frequency, reinforced that frequent responders do not rush through surveys. The data revealed response integrity among heavy responders. As would be expected, the very experienced panelists moved through the survey somewhat more quickly than new panelists. Their shorter completion times, however, appear to be the result of their greater experience and dexterity in navigating the survey page and instructions, not of speeding.
Frequent responders may be more in sync with actual consumer behavior
Our research shows some evidence that responses from frequent responders actually maybe more in line with real-world consumer behavior than responses from less frequent responders. Intent-to-buy scores are often overinflated in consumer survey research. Our research shows, however, that very experienced panelists have scores that are not as overinflated. Therefore, they appear to be more in synch with actual behavior.
This finding contradicts conventional wisdom that past participation is not desirable. That contradiction raises a key question. Are the scores from very experienced panelists more in line with reality for a good reason—they don’t have a desire to answer in an expected way, so they are more truthful? Or are their scores different for a bad reason—respondents want to get through the survey quickly and have discovered from their experience that answers indicating heavier usage and greater enthusiasm to buy often result in more follow up questions?
Our results indicate that the latter is not true. Experienced respondents are not trying to rush through the survey by under-reporting usage and intent-to-buy.
Gotta dig a bit more
In today’s environment, when consumers are faced with a daily barrage of hundreds or even thousands of marketing messages and survey opportunities, it may be time to re-think traditional ideas about what it means to be a frequent responder — and what effect past survey participation has on research results. Before “blacklisting” frequent survey takers, the industry needs to undertake further research and revise some potentially outdated assumptions about survey frequency, if the evidence continues to show it doesn’t affect quality.
While it’s important to be vigilant against fraud, research shows that real survey “cheaters” are a tiny minority of survey takers. In fact, the major causes of poor quality responses are fatigue, boredom, and ineffective survey design.
If you’re interested in Customer Experience topics, we think you might enjoy reading: Fit the product to the customer
There is much the industry can do to engage respondents
At SSI, for example, we’ve formed a focused respondent-experienced team that is dedicated to engaging and serving respondents, checking surveys from the point of view of the people taking them. Research proves that better survey design and keeping respondents happier directly improve data quality.
Respondents should be viewed as customers, just as companies who commission research are. They should be encouraged, consulted, engaged, and valued—not locked out for frequent participation.
About the Author: Chris DeAngelis is the Vice President of North American Sales for Survey Sampling International. SSI provides access to more than 6 million research respondents in 54 countries. Sources include SSI proprietary panel communities in 27 countries, a growing portfolio of managed affiliates, and our extensive global partner network.
At QuestionPro, we work on what we believe in. Schedule a demo and discover how to boost customer loyalty and collect valuable consumer insights through our CX survey and analytics management platform.
[…] The Frequent Responder Paradox (questionpro.com) […]