Why I’m Not Keen on Hypothesis tests for Normality and Homogeneity of Variance
Common analysis methods like ANOVA, linear regression and linear mixed models rely on the assumption that the residuals are approximately Normally distributed with constant variance. In a previous blog, we discussed how to assess residual diagnostic plots for compatibility with these two assumptions. However, some practitioners use hypothesis tests, such as the Shapiro-Wilk test or the Kolmogorov-Smirnov test for Normality, and Bartlett’s or Levene’s test for homogeneity of variance, to check these assumptions. I’m not a fan of this approach, and here’s why…
They test the wrong null hypothesis
When validating the Normality, our goal is to provide evidence supporting the Normality assumption. However, the null hypothesis of a Normality test assumes that the data follows a Normal distribution, so we can never conclude that our sample actually comes from a Normal distribution. Rather, if the p-value is sufficiently small, we reject the null hypothesis and conclude that the data is not Normally distributed. But if the p-value is large, can we claim that the data follows a Normal distribution? Unfortunately, not – because we can never accept the null hypothesis – we can only fail to reject it. A large p-value simply indicates a lack of evidence against the null hypothesis, and crucially, it doesn’t provide any evidence in support of it! To explore this important concept further, please refer to my previous blog Never accept the null hypothesis: It’s both wrong and dangerous.
The same argument holds for tests of homogeneity of variance. Since these tests assume the variance is equal across treatment groups, they cannot conclusively confirm homogeneity – they only provide evidence in favour of heterogeneity.
They can lack power
For small data sets, Normality and homogeneity of variance tests often lack power, meaning they may fail to detect important violations of these assumptions. As a result, even when there is non-Normality or heterogeneity, these tests may not gather enough evidence to reject the null hypothesis.

For example, consider the histogram of 20 data points shown here, which is right-skewed. Despite the data clearly being non-Normal, the Kolmogorov-Smirnov test gives a p-value of 0.265, providing no evidence against Normality.
They can be overly sensitive
Conversely, for large data sets, these tests can become too sensitive, detecting very minor, and potentially irrelevant, departures.
For example, consider the histogram of 1,000 data points, which were randomly generated from a Normal distribution with a mean of 1 and a variance of 0. The histogram is reasonably symmetrical and approximately bell-shaped. However, both the Kolmogorov-Smirnov test and the Shapiro-Wilk test return statistically significant p-values (at the 5% level) of 0.026 and 0.012, respectively; suggesting that the data is non-Normally distributed. Yet, in reality, with large data sets, methods like ANOVA, linear regression, and mixed models are quite robust to minor (and even moderate) departures from Normality, and a residual histogram like this one shouldn’t raise concerns

They don’t reveal potential outliers
An addition benefit of residual diagnostic plots is their ability to reveal potential outliers. Unfortunately, hypothesis tests don’t provide this insight, and outliers can lead to misleading conclusions about Normality or homogeneity of variance.
Consider the case where a data entry error has swapped over the treatment groups for two observations. In the residual diagnostic plots following ANOVA, these two outliers are immediately obvious, prompting further investigation. However, the results of Normality tests (e.g., a p-value of < 0.001 from the Shapiro-Wilk test) and homogeneity tests (e.g., a p-value of < 0.001 from Bartlett’s test) would lead us to incorrectly conclude that the residuals aren’t Normal and that there’s heterogeneity of variance between treatment groups, when in fact the underlying issue is a data entry error!

They can discourage critical thinking
Relying too heavily on these statistical tests can discourage critical thinking about the appropriateness of the assumptions underlying our analyses. For instance, concentration data is typically right-skewed, so we shouldn’t automatically assume it follows a Normal distribution. Indeed, concentration data often follows a log-Normal distribution and transforming it to a log scale can help satisfy the Normality assumption. Another example is when domain or expert knowledge suggests that one treatment group will exhibit more variability in its response than another. For instance, in a clinical trial investigating the effectiveness of a new medication to stabilize blood pressure, we may expect the placebo group to have more variability in blood pressure than the treatment group. In such a scenario, assuming constant variance is not appropriate. Thus, instead of relying on these tests to check for departures from normality and homogeneity, we should draw on existing knowledge to assess the validity of these assumptions.
They are often not applied to the correct random variable
ANOVA, linear regression, and linear mixed models assume that the residuals – not the response variable – are Normally distributed with constant variance. Unfortunately, some practitioners mistakenly apply Normality and homogeneity tests to the response variable instead of the residuals.
So, are these tests useful?
While I personally avoid using hypothesis tests to justify the residual assumptions of Normality and constant variance, they can still be useful in some cases. For reasonably large datasets (e.g., more than 30 observations per group), these tests can serve as a helpful alert for potential violations of these assumptions. However, I always recommend inspecting the residual plots before drawing any conclusions. Additionally, as mentioned above, non-significant results from these tests don’t necessarily indicate that everything is fine—it’s still essential to check the residual diagnostic plots to ensure the assumptions are reasonable. For small datasets, I recommend avoiding these tests altogether due to limited power to detect important violations of the assumptions, as well as the fact that a non-significant result doesn’t provide us with any meaningful information.