Non-parametric statistical tests are extremely flexible and do not require strong assumptions about the underlying data distribution (unlike parametric tests such as t-tests or F-tests). When the distributional assumptions of a parametric test are violated, the resulting p-values or confidence intervals can be inaccurate or misleading. Non-parametric tests provide a safer alternative in such cases, though they are often less powerful than parametric tests when the distributional assumptions actually hold.
However, that said, some parametric tests are relatively robust to minor, or even moderate, departures from their distributional assumptions. For example, ANOVA F-tests are fairly robust to violations of normality, especially with larger sample sizes and balanced designs.
Three common non-parametric approaches are randomization tests, permutation tests, and bootstrap tests. Let’s look at how they differ in purpose and methodology.
For illustration, let’s consider a simple example with a control group and a treatment group, where we want to test whether the group means differ. While this is a simple example, these methods can be applied to much more complex scenarios.
Randomization Test
Purpose: To test whether the observed difference in means from your experiment could plausibly occur by chance under the null hypothesis of no treatment effect.
How it works:
- Take your data and randomly reassign the treatment labels to your experimental units many times, whilst maintaining the original structure of your experimental design (e.g., in a randomized complete block design, the reassignment must preserve the block structure).
- For each randomization, calculate the difference in means.
- Compare your observed difference in means to the distribution of randomized differences. The further it lies in the tails, the stronger the evidence against the null hypothesis.
- Calculate the p-value under the null hypothesis as the proportion of randomized differences at least as extreme as the observed difference (for a two-sided test, count both tails).

Permutation Test
Purpose: To test whether the two groups come from the same distribution.
How it works:
- Pool all your data and randomly shuffle the group labels many times.
- For each permutation (i.e., shuffle), calculate the difference in means.
- Compare the observed difference to this permutation distribution.
- Calculate the p-value under the null as the proportion of permuted differences at least as extreme as the observed difference (for a two-sided test, count both tails).
But isn’t this just the same as a randomization test? Although randomization and permutation tests both rely on rearranging the data, they are subtly different. Randomization tests depend on the actual random assignment of treatments to experimental units, while permutation tests can also be applied to observational data. In the latter case, you’re assuming that the group labels are exchangeable under the null hypothesis: a modelling assumption rather than something guaranteed by a randomized experimental design.
Bootstrap Test
Purpose: To estimate the variability of a statistic and construct confidence intervals or estimate standard errors, especially when theoretical formulas are complicated or unknown. It can also be used to approximate p-values.
How it works:
To construct a confidence interval or estimate the standard error:
- Resample each group with replacement many times.
- For each resample, calculate the statistic (e.g., the difference in means).
- Construct the confidence interval (CI) using the relevant percentiles (e.g., 2.5% and 97.5% for a 95% CI) of the distribution of resampled differences.
- Estimate the standard error as the standard deviation of the distribution of resampled differences.

While the bootstrap is primarily valued for constructing confidence intervals and estimating standard errors (where its theoretical justification is strongest), it can also be adapted for hypothesis testing. However, this application is less straightforward and requires consideration and implementation of the null hypothesis.
To approximate a test of a null hypothesis:
- Pool all your data and resample with replacement as if under the null hypothesis.
- For each resample, calculate the test statistic (e.g., the difference in means)
- Compare the observed statistic to this empirical distribution.
- Calculate the p-value under the null hypothesis as the proportion of resampled differences that are as extreme or more extreme than the observed difference (counting both tails for a two-sided test).
A Note on Outliers
Bootstrapping naturally reflects the influence of outliers on a statistic. Because resampling is done with replacement, an outlier may be excluded, included once, or included multiple times in different bootstrap samples, producing a sampling distribution that captures the variability the outlier can introduce. In permutation and randomization tests, the outlier always appears exactly once, so while its value is fixed, its contribution to the test statistic varies only according to how the group labels are shuffled. As a result, bootstrap confidence intervals widen when outliers are present, reflecting the added uncertainty. Similarly, bootstrap p-values will reflect the variability introduced by the outlier through the resampling process. In contrast, permutation and randomization test p-values are determined by the fixed set of observations and how they are rearranged.
Key points:
- Randomization tests are tied to the actual random assignment of treatments in a designed experiment.
- Permutation tests are conceptually similar to randomization tests but can also be applied to observational data.
- In contrast, bootstrap methods focus on “how variable is my estimate?” but can be adapted for hypothesis testing with appropriate care.
How do I perform these tests in Genstat?
Genstat provides a variety of tools to perform randomization, permutation, and bootstrap analyses. The core functionality can be accessed via the menus. For example, to compare the means of two groups:
- Stats | Statistical Tests | Randomization Test for Two Groups
Description: Performs a randomization test of the hypothesis that two groups have equal means. - Stats | Statistical Tests | One- and Two-sample t-Tests > Options > Random Permutation Test
Description: Calculates the p-value for a two-sample t-test using a random permutation approach. - Stats | Bootstrap
Description: Performs bootstrap resampling to generate a distribution for a specified statistic, for example the difference between two means.
In addition, Genstat offers a range of more specialized randomization, permutation, and bootstrap methods for particular hypothesis tests and analyses. For example, for linear mixed models, there are permutation tests available for fixed and random effects.
Need a little extra support? Get in touch with the VSNi Team today to see how we can support you further.