As a survey designer, I am challenged to develop questions and response categories that maximize the information content for the client while minimizing the respondent’s cognitive load. This post I would like to talk about Point Allocation Method and how it compares to the rank order question type. The rank order question type is commonly used in survey research. It is easy to program and creates only a moderate amount of cognitive stress on the respondent (unless the researcher goes beyond 5 – 7 items). It is also one of the set of question types that is objective in nature and thus removes potential subjective bias.
However, the data it produces is ordinal which can present challenges for the analyst. Frequently, items have the same median ranking and on occasion, no item receives a top ranking, again based on the median. You can assign the “most preferred flag” to the item with the greatest response percent at either the top or bottom position, depending on how you phrased the question.
Point Allocation Method
As an alternative to the ranking question, the researcher can use a point allocation scale. In this scenario, respondents are asked to allocate 100 points across the slate of items for consideration. This produces interval-level data which can be averaged. It is interval because an item may receive no points or all of the points. It is also ready for analytical procedures that require numeric data, e.g. analysis of variance. Like the pairwise ranking problem, this question type is also objective and carries little risk for bias due to respondent subjectivity. The questions below show sample wording for both question formats.
Training providers can offer add-on features for a duration of time in order to enhance a student’s learning experience. With this in mind, please allocate 100 points across the course add-ons according to your perceived value as it relates to the learning experience. Allocate additional points to those add-ons which are more important to you. If an add-on has no importance it is okay to allocate zero (0) points to it.
• Unlimited access to certification practice exams
• Access to official vendor labs for hands-on practice
• On-demand access to searchable class recordings
• Unlimited course retakes across multiple delivery formats
• Certification exam voucher
• Digital courseware providing anytime/anywhere access to course materials
The rank order variation would substitute the follow language:
Training providers can offer add-on features for a duration of time in order to enhance a student’s learning experience. With this in mind, please rank the course add-ons according to your perceived value as they relate to the learning experience. Use a rank of one (1) as most important and six (6) as least important.
Both methods are in the domain of commonly accepted practice for survey research. Best practice for both variants is to limit the list to 5 – 7 items and to employ randomization (this minimizes first-order bias). The question type you choose will be a function of your client’s needs and your interest in analyzing the data.
