Pairwise Ranking and Pairwise Comparison

Pairwise Ranking, also known as Preference Ranking, is a ranking tool used to assign priorities to the multiple available options while Pairwise comparison, is a process of comparing alternatives in pairs to judge which entity is preferred over others or has a greater quantitative property.

Get Your Free Account Now

Join over
10 million users


Watch the 1-minute tour

What is Pairwise Ranking?

Pairwise Ranking, also known as Preference Ranking, is a ranking tool used to assign priorities to the multiple available options. It is primarily implemented to get insights about customer’s attitude, obtain feedback to learn about various customer perspectives and their decision-making capabilities.

In situations where there are numerous options and respondents might be overwhelmed while ranking those options, the use of Pairwise Ranking is suitable. It is not possible for humans to analytically rank several answer options according to true preferences. That is when this ranking tool comes into the picture and makes obtaining truest answers for survey questions convenient by making it suitable for respondents to answer without being confounded with the answer options.

All the options are directly compared to all the other options so that a list is created which consists of the most-preferred to the least-preferred options. The pairwise ranking process is a machine optimized procedure. Answer options are arranged orthogonality using a dynamic lookup model, i.e. all answers are bifurcated into groups of two and presented to the respondents. Respondents are expected to select one of these two options and are presented with 2 answer options until the end of answer options. Based on the answer options selected by the respondents, a machine based ranking algorithm selects the ranking of the provided options.

As and when respondents keep selecting their answer options, a tree is formed from all the answers. After the entire process is completed, the system creates a ranking list as per the received answers for the researchers to analyze.

This ranking tool is responsive and thus, favorable for respondents to answer online surveys by merely swiping left and right on the basis of their opinions without succumbing to the usually provided cumbersome answer options.

If there are N! ranking options, specifying ranks would require N log N number of data which suggests those are the required number of pairwise options.

What is Pairwise Comparison?

It’s not easy to come to a conclusion. Making decisions can be quite a task as it requires comparing alternatives with respect to a set of criteria. In case, there are more than two or more criteria, it becomes a difficult task to determine which criteria are more important.

To overcome this dilemma, one should be able to rank the criteria in order of importance and to be able to assign some relative ranking value to determine the degree of importance of each criterion. For example, planning a vacation. There might be different criteria like cost, time taken to reach the place, accommodation, number of locations to visit, quality of locations etc.

Pairwise comparison makes it easy to rank above criteria by comparing them in pairs. Let us first understand first what is pairwise comparison?


Pairwise comparison is a process of comparing alternatives in pairs to judge which entity is preferred over others or has a greater quantitative property. Pairwise comparison is one of the ways to determine how to access alternatives by providing an easy way to rate and rank decision-making. It is often used rank criteria in concept evaluation.

Pairwise comparison is used in conducting scientific studies, election polls, social choices etc. L.L Thurstone, famous psychometrician first introduced a scientific approach to pairwise ranking in 1927. Thurstone linked this approach to psychological theories designed by Ernst Weber and Gustav Fechner.

Pairwise Comparison Method

  1. Identify and arrange criteria to be ranked in a square matrix

    Select criteria that need to rank, compare and then arrange criteria in a square matrix. For this particular example let’s choose from product characteristics.

    A B C D E F
    Affordability A
    Maintenance B
    Safety C
    Functionality D
    Durability E
    Marketability F

    Each cell in the matrix corresponds to a comparison, hence the name “pairwise comparison”. Cells contain attributes that are important from a product criteria.

  2. Compare pairs of criteria across rows

    For each row, consider the criterion in the row with respect to each criterion in the rest of that row.

    So, the first comparison in the matrix is affordability and maintenance. Discuss within group and reach a conclusion for this question, which amongst the two attribute is more important. At this stage, this pair is the only one that will be compared. After comparing one pair, move on to the next pair. If the two criteria are equally important then, put both letters in corresponding cell. Note that the individual comparisons are pairwise – we completely ignore the other criteria.

    Say for example it is decided that affordability is more important than maintenance, then put A Say we decide that functionality is more important than durability. We would then put an A into cell 2, 4 of the matrix. Continue doing this till all the empty cells have been filled.

    For example, once all the pairs are compared this is how the matrix will look

    A B C D E F
    Affordability A - A C D AF A
    Maintenance B - - C D B B
    Safety C - - - D C F
    Functionality D - - - - D F
    Durability E - - - - - F
    Marketability F - - - - - -

    Two alphabets in the same cell indicate that the two criteria are equally important.

  3. Create Ranking

    Create a sorted list of criteria, ranking is based on number of cells containing their corresponding letter. For example, affordability is marked with letter A so the value is 3 because there are 4 A in matrix. Therefore,

    • Maintenance: 2
    • Safety: 1
    • Functionality: 4
    • Durability: 0
    • Marketability: 4

  4. Assign Weights

    There are two criteria that need to be considered while assigning weight:

    • The total of all the weights must be 100%.
    • The weights must obey the qualitative ranking given above.

    One very easy way to get that initial set of values is to assume a linear proportion between all the weights and solve the following equation:

    100= 2x + 1x + 4x + 0x + 4x

    x= 9.09 (approximately)

    Here, coefficients are the number of occurrences of each criterion in the matrix. Therefore,

    • Maintenance: 2x =18.18%
    • Safety: 1x = 9.09%
    • Functionability: 4x = 36.36%
    • Durability: 0x = 0.01%
    • Marketability: 4x = 36.36%

    0.01 for durability is obtained by round off all the other calculations.

    From the above calculations the ranks for attributes in a product would be:

    • Rank 1: Functionality and Marketability
    • Rank 2: Maintenance
    • Rank 3: Safety
    • Rank 4: Durability

How to Implement QuestionPro as a Pairwise Ranking and Comparison Tool?

Pairwise ranking can be configured using QuestionPro Workforce. A new feature called the “Path Finder” has been added under Workforce where a researcher can add Pairwise ranking questions along with answer options.

Click on the Add a New Path Finder option >> After entering the name, question and the answer options and once the inputs are saved, they will appear in a manner similar to this:


The analytics tab will show the researcher real-time dashboard about the respondent rankings along with a word cloud and leaderboard.

Average Rank: The answer option having the highest average ranking score is considered to be the preferred options among respondents.

This is the average pairwise ranking calculation, where:

w = Ranked position weight

x = Number of responses for an answer answer option

x1w1 + x2w2 + x3w3 ... xnwn/Total

Weights are the inverse of respondents chosen answer option ranks, i.e., if a respondent assigns rank 1 will have the highest weight while the option they select to be the last rank will have the lowest weight. These are the default weights which can never be edited.

For example, if a Ranking question has 3 answer options, these are the assigned weights:

The #1 choice has a weight of 3

The #2 choice has a weight of 2

The #3 choice has a weight of 1

Word clouds: Also called text clouds or tag clouds, where the answer options with higher average weight, i.e., better ranking, will appear in bigger and bolder fonts in the word cloud.

Leaderboard: Leaderboard shows most preferred response alternative which receives rank 1 and it shows the number of times that particular option has been chosen. Upon clicking on the option, it will show a list of respondents who chose that answer option.

New Dashboard

Two answer options at once will be shown to individuals for selection. After responding to all the questions, this is how the results will be displayed:

Take survey

Ranking order

Get Your Free Account Now