Dichotomous questions – Are they useful?

Are dichotomous questions useful in market research? The answer is both yes and no.

By definition, a dichotomy has two parts. In the framework of survey design, dichotomous questions have two possible answer choices. The most common being the Yes/No dichotomy.

Other dichotomy questions examples include:

• True/False
• Male/Female
• Up to 45/45 and over

This question framework is appropriate for factual reporting, but can be used inadvertently in a leading manner.

The benefits of dichotomous questions are two-fold – they are easy to comprehend as well as they are short. Anytime we can simplify the survey experience, the user (respondent) wins and this can lead to greater survey completion rates. Still wondering – What is a Dichotomous Question?

But what about the cons of Dichotomous questions?

The ease and simplicity of dichotomous questions inherently place limits on the analysis that can be performed. If we ask a respondent if they have used our product in the last six months (Yes/No), we have only two groups to analyze. If we re-frame the question as below, then we have more options.

How many times in the last six months have you used our product? If you have not used the product please write-in zero (0) ____ [open-ended numerical response]

Post-facto in the analysis phase we can look at average product usage by the group. As analysts, we can always collapse data down (0 = non-user and 1+ = user), however, we cannot expand if we first ask a dichotomous question.

Below are examples of two ways to enquire about the perceived impacts of a menu change.

1.    Do you think our new menu additions have improved the dining experience at La Grange?  Yes/No
2.    In your opinion, how have the menu changes at La Grange impacted your dining experience?

Options can be:

• Much Worse
• Worse
• No Change
• Improved
• Much Improved

The first question is clearly leading as it sets the respondent’s mind to focus on improvement. Respondents want to agree with us (acquiescence bias) therefore it is critical we don’t lead them down the wrong path. The second option allows us to measure direction (worse to improved) and intensity. It also provides a neutral option for those who feel there has been no change due to the menu additions.

Note the previous thought about collapsing data, you can take responses from the second question and collapse them into groups: those who think the change has been for the worse vs. those who think the change was positive, thus achieving a dichotomy.

Dichotomous questions offer the respondent an easy user experience, however, they limit what can be done with the data. In most cases, we can open up the number of categories and then collapse them if necessary in the analysis phase, e.g. top box vs. bottom box, thus allowing us more granularity.

This is a path I would normally recommend, with the exception of surveying of children, where dichotomous questions can be the most effective format. There are many factors to consider when developing survey questions – we must always strive to balance respondent experience with the need to collect data which helps us make solid business decisions.