ANOVA Analysis in QuestionPro BI

QuestionPro BI delivers built-in ANOVA (Analysis of Variance), so teams know whether differences in scores across groups are real or due to chance before they act on them.

• One-way ANOVA: Compares the mean of a measure across three or more groups of one variable, such as satisfaction by region.
• Two-way ANOVA: Tests two grouping variables at once and the interaction between them, such as satisfaction by region and customer type.

Every result includes group means, the F-statistic, degrees of freedom, and the p-value, ready to share with stakeholders.

The test is matched to the measurement level of the variables being compared:

• Nominal or ordinal by nominal: Chi-square test, or Fisher's exact test for small samples.
• Interval/ratio by one variable with two groups: t-test.
• Interval/ratio by one variable with three or more groups: One-way ANOVA.
• Interval/ratio by two grouping variables: Two-way ANOVA.

Rating scales, numeric inputs, and sliders are analyzed as interval/ratio measures. Single-select and multi-select questions serve as nominal groupings.

1. Go to Reports and click Create report.

2. Select Crosstab report, enter a name, select your data source, and click Create.

3. Add the numeric measure, such as a satisfaction rating, to the rows.

4. Add the grouping variable, such as region, to the columns. For two-way ANOVA, add a second grouping variable, such as customer type.

5. Click Settings in the top-right corner and open the Analytics tab.

6. Select ANOVA and choose the confidence level.

7. Click Apply. The ANOVA results appear with the report.

• Group means: The average of the measure for each group.
• F-statistic: The ratio of variation between groups to variation within groups. Higher values signal stronger differences.
• Degrees of freedom: Between groups (number of groups minus one) and within groups (total responses minus number of groups).
• p-value: A p-value below the significance threshold (0.05 at 95% confidence) confirms that at least one group mean differs significantly.

Note: A significant result confirms that the groups differ. Compare the group means to pinpoint where.

Two-way ANOVA reports three effects, each with its own F-statistic, degrees of freedom, and p-value:

• Main effect of the first variable: Whether the mean differs across its groups, averaged over the second variable.
• Main effect of the second variable: Whether the mean differs across its groups, averaged over the first variable.
• Interaction effect: Whether the effect of one variable depends on the level of the other.

Note: Read the interaction effect first. When it is significant, interpret each main effect within the levels of the other variable.

• Independence: Each response belongs to one group, or one group combination in two-way ANOVA.
• Normality: The measure is approximately normal within each group. ANOVA is robust to moderate departures when groups contain 30 or more responses.
• Homogeneity of variance: The spread of the measure is similar across groups. Balanced group sizes strengthen this.
• Sample size: Each group, and each group combination in two-way ANOVA, contains enough responses for a stable mean.

Note: Apply filters to exclude incomplete or invalid responses before running the test.

When should I use ANOVA instead of a t-test?

Use a t-test for two groups and ANOVA for three or more. ANOVA tests all groups in one comparison, avoiding the false positives that multiple t-tests create.

When should I use two-way ANOVA instead of one-way ANOVA?

Use two-way ANOVA when two factors may influence the measure, or when one factor may change the effect of the other. Separate one-way tests cannot detect that interaction.

Can I run ANOVA on categorical questions?

Categorical questions serve as grouping variables; the measure being compared is numeric, such as a rating or score. To test association between two categorical questions, use the Chi-square test.

What confidence level should I use?

95% confidence (a p-value below 0.05) is the standard for business and academic research. Use 99% when the cost of a false positive is high.