What Does Quality Really Mean in Market Research

imageI’ve been doing some research on market research panels and noticed the overwhelming number the the word “quality” appeared in articles and blog posts.  This experience reminded me of doing customer satisfaction research for manufacturers and getting into long drawn out battles about why “price” and “quality” weren’t real reasons for buying and not buying because they were too ambiguous.

Opportunities for Quality inside the Market Research Process

Let me explain what I mean by “quality is ambiguous.”  In the world of manufacturing quality isn’t a step — it runs throughout the process.  When we measure quality — we’re actually measuring the quality of the customers’ experience with our process and our product or service.  So quality happens at any stage of the process.

Here’s an example – Let’s say that you receive a catalog in the mail.  The catalog has all kinds of products and as you leaf through it, you see something you like – you’re ready to order — but WAIT — there’s no phone number, no web site, NOTHING!  Notice that this impacts the quality of your experience.  You want to order, but something is missing that’s making it difficult or impossible for you to complete your task.  In a way, it’s the same as if a button on your phone didn’t work – you set out to dial, but the quality of the part failed.

My point here is that quality is something that happens at many steps of the process.

Quality in market research

Let’s get back to quality in market research.  I’m researching online survey panels and notice that (as expected) panel quality is a real concern.  After all, when you purchase panel responses to do your branding research you are making the assumption that people are responding honestly to your survey.  But what’s really nagging at you is whether the people who are responding are:

  • really who they say they are
  • responding because they care about the survey — or responding because they want to incentive
These are all valid concerns about panels and panelists and there is certainly a quality process that needs to happen to assure that these concerns are alleviated.  
Another potential quality touch point is the actual online survey.  Something as subtle as how questions are worded and the choices given can widely impact the quality of the data received.  This is perhaps the one area of quality that professional market researchers are most upset about with the onset of DIY market research — fewer and fewer people are consulting professional researchers about the quality of the survey instrument.  And because of this — the quality and predictability of the results is in questions.
Another aspect of market research quality lies in data analysis.  Looking at the data and understanding what it really means.  Even as a DIY market researcher myself, I often consult with professionals to double-check my results.  A data chart can say one thing – but professional data analysts and researchers know how to see inconsistencies in your data and can really provide valuable feedback when it comes to understanding what the data says and what it doesn’t say —
True story —
I conducted a market research project where a question on the survey read “Would you like to receive “Product X” in a sterilized package?”  the answer was overwhelmingly “Yes” .  The question that wasn’t asked was “Would you pay 20% more for product X?”  In other words, you would be on thin ice making changes to a product based on a “Would you LIKE” question.  If you asked me “Would you like a Mercedes?”  I might say “yes.”  If you asked me “Are you going to buy a Mercedes this year?”  I would say “No” — these are two different questions.
Quality can be a tricky word.  It encompasses so much more in reality than we can imagine.  As you start your next online survey project, pay attention to all aspects of quality in terms of your customers experience and YOURs.