Further inference from an ANOVA table: residual variance to standard errors and confidence intervals

Below is an example of a 2-way Analysis of Variance (ANOVA) for a randomised complete block design. From the ANOVA table, we’re going to see how to calculate: 

• The standard error of a mean (SEM). 
• The confidence interval (CI) for a mean.
• The standard error of the difference between two means (SED).
• The least significant difference between two means (LSD).
• The confidence interval for the difference between two means.

Understanding residual mean square

The ANOVA table gives us an estimate of the residual mean square () – also known as the mean square error, residual error or residual variation. This is the variation in the data unexplained by the ANOVA model.  In the example above, this is the variation remaining after the block effects (block) and treatment effects (Nitrogen, Sulphur and the Nitrogen by Sulphur interaction, Nitrogen.Sulphur) have been accounted for.   

Calculating standard error of a mean (sem)

The standard error of a mean (SEM) is calculated using the following formula:

where n is the number of replicates (or sample size).

The SEM describes the uncertainty in our estimate of the population mean for a treatment from the available data. The bigger the sample size (i.e., the larger n), the smaller the SEM and the more precise our estimate is. 

In our example, there are 12 unique treatment combinations: the 3 levels of Nitrogen by the 4 levels of Sulphur. Note, we can obtain the number of levels of each treatment, and number of blocks, from the degrees of freedom in the ANOVA table.

Furthermore, the total number of degrees of freedom + 1, gives us the number of experimental units.  Thus, in our example, there are 36 experimental units, with each of the 12 unique treatment combinations occurring exactly once in each of the 3 blocks. 

Therefore…

• The 12 Nitrogen by Sulphur means have replication of 3 (1 replicate per block).

For example, the 3 replicates of the treatment corresponding to the first level of Nitrogen and the second level of Sulphur (Nitrogen 1 Sulphur 2) are highlighted yellow in the schematic below of our randomised complete block design:

Thus, the standard error for the Nitrogen by Sulphur means is:

• The 3 Nitrogen means, pooled over the four Sulphur levels, have replication of 12 (3 blocks x 4 levels of Sulphur).

For example, the replicates of the first level of Nitrogen (Nitrogen 1) are highlighted in yellow:

(Note, within each of the 3 blocks, a given level of Nitrogen corresponds to 4 unique treatment combinations: 1 at each level of Sulphur). 

Thus, the standard error for the Nitrogen means is:

• The 4 Sulphur means, pooled over the 3 Nitrogen levels, have replication of 9 (3 blocks x 3 levels of Nitrogen).

For example, the replicates of the second level of Sulphur (Sulphur 2) are highlighted yellow:

Thus, the standard error for the Sulphur means is:

Confidence interval (ci) for a mean

The confidence interval (CI) for a mean is

      x

where is the critical value of the  distribution with degrees of freedom. For a confidence interval of C%, . For example, for a 95% confidence interval, . The refers to the residual degrees of freedom. This can be read directly from the ANOVA table.

Interpreting C% confidence interval

In a nutshell, a C% confidence interval for a mean is a range of values that you can be C% certain contains the true population mean. Although, strictly speaking, the confidence level C% represents a long-run percentage: the C% confidence interval gives an estimated range of values that we would expect the true, but unknown, population parameter to lie within C% of the times, should we repeat our experiment a large number of times. 

The tables of means for our example are given below:

For example, the 95% confidence interval for the overall mean for Sulphur level 4 is:

      x

      x

Similarly, the 99% confidence interval for the Nitrogen level 1, Sulphur level 3 mean is:

      x

      x

Calculating standard error of the difference (sed)

The standard error of the difference between two means (SED) is calculated using the following formula:

Note: The formula is different when the sample sizes of the means being compared are unequal.

The SED describes the uncertainty in our estimate of the difference between two population means. 

For our example, the SED between …

 a) two Nitrogen by Sulphur means is:

b) two overall Nitrogen means is:

and

c) two overall Sulphur means is:

Using LSD for statistical significance

Two means can be compared using their least significant difference (LSD). The LSD gives the smallest value in which the absolute difference between the two means is deemed to be is statistically significant at the α level of significance. The LSD is given by:

     ( x 100%) x

Thus, for our example, the LSD(5%) for comparing …

a)  two Nitrogen by Sulphur means is:

     % = x

                                                      = x

                                                      =

b)  two overall Nitrogen means is:

     % = x

                                         = x

                                         =

c)  two overall Sulphur means is:

     % = x

                                         = x

                                         =

Using the overall means for Nitrogen as an example, at the 5% significance level…

• there is statistical evidence that the Nitrogen 1 and Nitrogen 2 means differ

, the LSD(5%)

• there is statistical evidence that the Nitrogen 1 and Nitrogen 3 means differ

, the LSD(5%)

• there is NO statistical evidence that the Nitrogen 2 and Nitrogen 3 means differ

, the LSD(5%)

Confidence interval for difference between means

The difference between two means can be also compared, and more fully described, using a confidence interval for the difference between two means. This is given by:

x

or, equivalently,

x x 100%

where and are the two means being compared.

Once again, using the overall means for Nitrogen as an example, the 95% confidence interval for the difference between:

• the Nitrogen 1 and Nitrogen 2 means is:

• the Nitrogen 1 and Nitrogen 3 means is:

• the Nitrogen 2 and Nitrogen 3 means is:

Notice that the CIs comparing Nitrogen 1 with Nitrogen 2, and Nitrogen 1 with Nitrogen 3, both exclude zero. Hence, we can conclude, at the 5% significance level, that the mean for Nitrogen 1 is significant differently from both the Nitrogen 2 and Nitrogen 3 means. (In this case the mean for Nitrogen 1 is lower than that of Nitrogen 2 and Nitrogen 3). Conversely, as the CI comparing Nitrogen 2 and Nitrogen 3 includes zero, we conclude that there is no evidence of a difference between these two means (at the 5% significance level).

Luckily for us, we rarely need to calculate these quantities ourselves, as they are generated by most statistical software packages. However, it is useful to understand how they are calculated and how they are related. For example, in order to scrutinize reported results, or to calculate, at a later date, a quantity that you’ve forgotten to generate.

ANOVA tools in Genstat

Genstat has a very powerful set of ANOVA tools, that are straightforward and easy to use. In addition to the ANOVA table, you can readily output the treatment means, SEMs, SEDs, LSDs and CIs.

About the author

Dr. Vanessa Cave is an applied statistician interested in the application of statistics to the biosciences, in particular agriculture and ecology, and is a developer of the Genstat statistical software package. She has over 15 years of experience collaborating with scientists, using statistics to solve real-world problems.  Vanessa provides expertise on experiment and survey design, data collection and management, statistical analysis, and the interpretation of statistical findings. Her interests include statistical consultancy, mixed models, multivariate methods, statistical ecology, statistical graphics and data visualisation, and the statistical challenges related to digital agriculture.

Vanessa is currently President of the Australasian Region of the International Biometric Society, past-President of the New Zealand Statistical Association, an Associate Editor for the Agronomy Journal, on the Editorial Board of The New Zealand Veterinary Journal and an honorary academic at the University of Auckland. She has a PhD in statistics from the University of St Andrew.