The Multiple Comparison Problem

For the majority of statistical analyses, we find ourselves with the Multiple Comparison Problem. This is the case where we focus on statistical inference for several hypotheses simultaneously. For example, we have t = 5 treatments to compare against each other, then there is a total of k = 10 tests to perform. With so many tests, we are likely to find by chance something really different test one significant test (a false positive). This is where we might want to correct our testing process for multiple comparisons, usually by considering a stricter (lower) significance level but this has the disadvantage of reducing our statistical power to detect some real differences! Hence, what do we do? In these notes we will discuss some of the elements associated with this problem and we will provide recommendations on how we address this issue.

First, let’s refresh the meaning of significance level. If you want more detail, this was discussed previously in another blog. A significance level refers to a value, say α = 0.05, that we use to decide if we should reject a null hypothesis (e.g., no difference in the means between two treatments). This α value represents the proportion of times (here 5%) that we expect to reject the null hypothesis when in fact it is true. The issue with multiple comparisons is that we are now performing, say k comparisons (10 in our case, calculated as k = t × (t – 1)/2), so the chance of at least one mistake is very high. The formula for the calculation of this mistake, expressed as $\boldsymbol{\alpha_f}$ (the experiment-wise significance level) is:

$\large \boldsymbol{\alpha_f = 1 – (1 – \alpha)^k}$

$\large \boldsymbol{\alpha_f = 1 – (1 – 0.05)^{10} = 0.401}$

Hence, with k = 10 tests, there’s a 40% chance that we will incorrectly declare one or more pairs of means to be different. If we had a total of t = 15 treatments to compare then we will need to do k = 105 comparisons, for which the probability of at least one mistake is 0.995!

As you can see, this problem is concerning. In the following sections, we will describe the range of options from doing nothing to a full experiment-wise protection. But keep in mind that these are not mutually exclusive, as some analyses might be approached with more than one option.

1. Continue with our original significance level

One option is to continue using our original significance level, say α = 0.05 for each of the comparisons (that is α = α_f). However, most importantly, we have to recognise that this is likely to result in false positives. Hence, depending on the magnitude of k, we should expect to report some excess of significant results. As long as we are aware of this, and we make this explicit when reporting our analysis, then this shouldn’t be much of a problem. Note that if the objective of the study is discovery, then having some false positives that will later be re-assessed in further studies/analyses (i.e., additional testing with fewer treatments) is not a bad thing.

2. Use an experiment-wise significance level

Let’s move to the other extreme. In this case we will calculate an appropriate α_f (the experiment-wise significance level) based on k tests. This means that for ALL tests we are going to perform, we set a significance level for an individual test ${(\alpha_b)}$ that will result in an $\alpha_f$ x 100% error rate (e.g., 5%) in the complete study. The calculation for this is based on the famous Bonferroni correction:

$\large \boldsymbol{\alpha_b = (1 – \alpha_f)^{1k} \sim \alpha_f / k}$

If we set our $\alpha_f$ to be 0.05 (5%) then the $\alpha_b$ for a single test changes to 0.0050 and 0.0005 for multiple comparisons on 10 and 15 treatments, respectively.

As you can see, now it is a lot harder to report significant results for single test, but on the other hand, we are being careful not to report too many false positives in the complete experiment. This is a conservative approach, but one of its main costs is that when the number of treatments, t, is large then in many cases the α is so small that nothing is significant! This may be what we want for protection against false positives, but it also might result in no true differences (or effects) being identified as we are limiting our statistical power greatly.

In order to avoid the above extremes, there are several control methods available, which are often known as post-hoc tests. You’ve probably have heard of Duncan, Tukey, Scheffe, Dunnett and many other multiple comparison methods that are often reported. They all try to achieve a compromise between no control and full control.

The issue is then which test to use (unless this is chosen for you). This is a real decision but most statistical software gives you options to request many of them simultaneously (for good or bad). However the arbitrariness of the decision on which test to use is always going to be present and this can lead to conflict with other users (and particularly reviewers). In any case the use of these tests should give you reasonable compromise and some peace of mind!

When planning your study you define clearly what your hypotheses of interest are and these correspond to a relatively small number. Then, it is understood that you do not need any correction of your significance level. The key here is that 1) your focus is on just those pre-planned comparisons 2) these are an integral part of your experimental design and 3) you will not perform additional comparisons (or hypothesis tests) after exploring your data. This will ensure that you maintain the level of false positive rates stated at the beginning of your study (for which you decided a reasonable level of α beforehand).

Let’s illustrate this with an example: if the main objective of your study is to compare your new treatment formulations against a commercial control then you have only t-1 comparisons of interest. For example, with t = 5 we will do 4 comparisons instead of 15. As each of these pre-planned comparisons represent your real hypotheses of interest, you can always use your predefined significance level with no adjustment. Here you are saying that you are willing to report the comparison of each new formulation against the commercial control with only α × 100% (e.g., 5%) expected false positives. This logic is based on a probabilistic plan of performing a carefully planned experiment (including replication and power) with only few reasonable comparisons.

In the above example there’s no pre-planned interest in comparing each treatment against each other so why do that and risk reporting many false positives. Here, you have to remember: the more times you gamble the more often you will lose.

An important aspect that we have not mentioned is when the treatments have a structure (for example, a two-way factorial arrangement). In this case, multiple comparisons are inappropriate, as we should focus on evaluating the main effects and their interactions instead of brute force comparisons. Something similar happens when we have a factor that is better treated as a continuous covariate providing a different (and often better) insight on the dynamics of this effect.

One element to be careful of is that most statistical software will allow you to implement, in just seconds, several of the multiple comparison tests. This is bad practice as you end up picking the method that accommodates your pre-conception of the results of the study (and trying to justify it afterward). In order to avoid this, it is much better to have the testing method decisions made during the planning stage of your study.

As you have heard, the key for any of these successes is the specification of the hypotheses of interest before you do the analyses, and even better before you start the experiment. At that moment where the level of risk you are willing to commit is decided upon (and note that that does not have to be the traditional 0.1, 0.05 and 0.10 levels). Here, you determine this risk according to the study objective, the stage of research and the current and potential future uses of the study results.

Finally, you also have to consider that your decision to control or not to control the multiple comparison experiment-wise significance level will likely create some conflict or disagreement between you, other scientists, reviewers and/or your target audience. In order to avoid this, the always recommended approach is you report all results. This includes treatment averages (or differences), the size of the effect, a measure of variability and the full set of p-values for the respective tests. This will help the reader to decide those results are useful evidence for them and not just for you.