Demystifying the P-Value: Understanding its Meaning and Application

Demystifying the p-value: Understanding its meaning and application

Dr. Vanessa Cave

21 June 2023

In statistical analysis, the p-value serves as a valuable tool for determining the strength of evidence against a null hypothesis. It plays a crucial role in statistical inference, particularly in the realm of scientific research. This blog aims to unravel the true meaning of the p-value, explore its interpretation, and shed light on its appropriate usage.

Deciphering the p-value's meaning

The p-value represents the probability of obtaining a test statistic as extreme as, or more extreme, than the one observed under the assumption that the null hypothesis is true. It exists on a continuum between 0 and 1 and serves as a measure of the strength of evidence against the null hypothesis. Importantly, the p-value does not determine the probability of the alternative hypothesis being true.

Interpreting the p-value

The magnitude of the p-value indicates the strength of evidence against the null hypothesis. A smaller p-value suggests stronger evidence for rejecting the null hypothesis. However, defining what constitutes a small p-value is not straightforward1. Commonly adopted guidelines suggest p < 0.001 as very strong evidence, p < 0.01 as strong evidence, p < 0.05 as moderate evidence, p < 0.1 as weak evidence or a trend, and p ≥ 0.1 as insufficient evidence. Nonetheless, it is important to avoid binary categorisation of p-values as significant or non-significant based solely on arbitrary thresholds.

Role of effect size and confidence intervals

To draw meaningful conclusions from statistical analyses, it is essential to consider the size of the effect in conjunction with p-values. Confidence intervals are often used to describe the effect size and the precision of its estimate2. It is important to note that statistical significance does not necessarily imply practical or biological significance. Small p-values can arise from large sample sizes and small effects or small sample sizes and large effects.

Importance of sample size

The p-value is influenced by the sample size. A very large sample size may lead to the rejection of the null hypothesis, even with a minuscule effect size, and even if the null hypothesis is, to all intents and purposes, essentially true. Conversely, with a very small sample size, it may be exceedingly difficult to reject the null hypothesis, even if an substantial effect size is observed. Thus, the interpretation of p-values should consider the size of the study.

Practical considerations for p-value interpretation

It is crucial to remember that a p-value quantifies evidence against the null hypothesis, but it does not provide evidence in support of the null hypothesis. Accepting the null hypothesis solely based on a large p-value is a fallacy that should be avoided3. Additionally, p-values depend on the underlying assumptions and statistical methods employed in the analysis, emphasising the importance of robust and appropriate statistical practices.

Drawing meaningful conclusions

The p-value serves as a statistical tool for assessing the strength of evidence against the null hypothesis. It plays a critical role in statistical inference, particularly in biological research. By understanding its true meaning, interpreting it alongside the effect size (say, using a confidence interval), and considering the impact of sample size, researchers can draw meaningful conclusions from their statistical analyses. Emphasising the appropriate usage and avoiding binary categorisation, the p-value remains a valuable component of statistical reasoning.

Related blogs:

About the author

Dr. Vanessa Cave is an applied statistician interested in the application of statistics to the biosciences, in particular agriculture and ecology, and is a developer of the Genstat statistical software package. She has over 15 years of experience collaborating with scientists, using statistics to solve real-world problems.  Vanessa provides expertise on experiment and survey design, data collection and management, statistical analysis, and the interpretation of statistical findings. Her interests include statistical consultancy, mixed models, multivariate methods, statistical ecology, statistical graphics and data visualisation, and the statistical challenges related to digital agriculture.

Vanessa is a past President of both the Australasian Region of the International Biometric Society and the New Zealand Statistical Association, on the Editorial Board of The New Zealand Veterinary Journal and an honorary academic at the University of Auckland. She has a PhD in statistics from the University of St Andrew.