My last article about changing rating scales prompted this e-mail from a reader.
Over many years of surveying customers and guests I have had to grapple many times with changing methodology: not just changing scales but in changing data collection methods. Now, more than ever, many clients are thinking about, if not already, switching to data collection through email surveys and internet based surveying. As the saying goes “Everyone else is doing it so we should too”.
What faces us is the question of how to explain the differences in scores achieved by the surveying methods. For example, having conducted large scale guest satisfaction surveys across many European countries using intercept surveying we must now convert our collection methods to internet services. Using the exact same questionnaire but administering the survey by email we witness an average drop in satisfaction rating of 9-10% across the board. For companies firmly focused on customer satisfaction this drop is very disconcerting.
Yes, the “halo effect” may explain some of the change but a drop of 10%? Does primacy and recency affect the scores but by how much? Having intercepted the guest before leaving the premises we now mail them a day or two later and get the survey completed sometimes up to 5 to 7 days later. Does the level of satisfaction just drop off as time passes?
In previous years we have tested the change in methodology between face to face, postal and telephone interview and can clearly see that the face to face scores are always the highest, not only because we set the parameters on how respondents can be selected and can to some extent control all the environmental variables, but once we lose this control the data changes substantially.
I just want to pose the question to other interested marketers and ask how they understand this variance in scores, if in evidence and how best can it be explained.
— Madeleine O’Carroll, consultant researcher working in the EMEA region, specialising in hospitality research.
I’ve done some looking on the internet, but haven’t found a lot of information on this topic. What’s been your experience? Does anyone have resource or advice for Madeleine? Leave a comment here or send me any links or resources that you have and I will post them.
Better yet – if you’ve had experience on this and would like to send a guest post on this topic – please e-mail it to me ivana@thirdforce.net and I will post it. Be sure to include a one-sentence bio and I link where I can send our readers for more information about you and your organization.
