When designing surveys, we often need to decide whether to use an open ended question versus close ended questions to get specific information. Yet we need to be aware of the fact that open ended question and close ended questions each have their own strengths and weaknesses and perform in different ways.
In a study conducted by the Pew Research in their November 2008 post-election survey, respondents were asked, “What one issue mattered most to you in deciding how you voted for president?” One group was asked this question in a close ended questions format while the other group was asked this question in an open ended question format. The results are displayed below:
In the close ended questions format, 58% of respondents chose “The economy,” while in the open-ended format only 35% wrote in a response that indicated “The economy.” Note also that in the close ended questions format, only 8% of respondents selected “Other,” while in the open-ended format 43% of respondents wrote in a response that would have been categorized as “Other.”
Open Ended Question and Close Ended Questions
1. The open ended question and close ended questions are different tasks for respondents. In the open-ended task, respondents write down what is readily available in their minds, whereas in the close ended questions task we have respondents focus their “attention on specific responses chosen by the investigator,” (Converse and Presser, 1986).
2. Asking the same question in these two different formats will almost always produce different results. Many investigators have demonstrated this over several decades.
3. Few respondents are going to select the “Other” category and write in responses that are different than the answer choices that are listed.
So what does this mean for us? If you can, do qualitative research first and make sure your close ended questions represent the items that are in people’s heads. We need the list of items to be complete since few respondents will select the “Other” category. It may also be necessary to list items that are not readily available to respondents if those items are important to you. I will discuss this further in my next blog post.
Consider doing split sample studies where half of the respondents see the question in the open-ended format while the other half sees the question in the close ended questions format.
When presenting results, I have found it helpful to explain to the audience, in a sentence or two, the fundamental differences between open ended question and close ended questions. This helps them understand that these are not necessarily precise measurements, but measurements that require some interpretation relative to other questions in the survey and other information from qualitative research. Hence, that is why they need an analyst like you or me!
The differences in the performance and appropriate use of open ended question and close ended questions are an enormous topic. I will follow up with more discussion in my next few blog posts.
* Converse, Jean M., and Stanley Presser. 1986. Survey Questions: Handcrafting the Standardized Questionnaire, Beverly Hills, CA: Sage.
David F. Harris designs qualitative and quantitative research for companies that want better information for decision-making. He also conducts state-of-the-art training on questionnaire design. He is the author of, The Complete Guide to Writing Questionnaires: How to Get Better Information for Better Decisions, 2014. He received his B.A. from Reed College, in Portland, Oregon, and his M.A. in Quantitative Psychology from the L. L. Thurstone Psychometric Laboratory at the University of North Carolina at Chapel Hill.
