Analysis of Censored Data

Censoring is a common problem faced when analysing experimental data. It occurs during data collection when measurements cannot, or are not, taken above or below a bound. For example, left-censoring is a typical feature of chemical concentration data due to the lower limit of quantification. This is the lowest concentration that can be reliably measured by the measurement procedure. Concentrations that fall below this limit will therefore be left-censored. This means we don’t know the true concentration, only that it is less than the limit of quantification.
A nice example of right-censored data was recently encountered by my colleague, David, when analysing the growth of bacterial isolates on agar plates. Growth was measured by the area of the agar plate covered by the bacteria. However, some bacteria grew so well that the entire agar plate was covered at the conclusion of the experiment, resulting in right-censoring. Another example was in a wheat trial, where some grain was spilled from several bags containing the harvested plot yields, and so the yield was known to be above the recorded weight for these plots.

Despite censoring posing a ubiquitous problem in the analysis of experimental data, how to robustly analyse censored data is a challenge! Often ad hoc solutions are resorted to, such as replacing left-censored observations with half the limit of quantification, or the censoring is ignored altogether – both of which are unsatisfactory.
To provide you with the tools you need to robustly analyse your censored data, the Genstat team have implemented a suite of specialised facilities. These include the ability to analyse censored Normal data using an ANOVA or a linear mixed model (LMM), as well as censored Poisson, gamma or negative binomial data using a generalized linear model (GLM), generalized linear mixed model (GLMM) or a hierarchical generalized linear model (HGLM).
Example: Censored Cannabidiol Data
An experiment was conducted to compare the mean cannabidiol (CBD) potency of three cultivars nourished with two different nutrient feeds: the standard feed and a new feed. The experiment was laid-out as a split-plot, with nutrient feeds applied to the whole-plots (here, rows), and with 12 replicates.
At harvest, the percentage of CBD relative to plant wet weight (i.e., the CBD potency) was measured using an analytical procedure that had a lower limit of quantification of 2% and upper limit of 15%. Unfortunately, some plants had potencies less than the lower limit, resulting in left-censoring, whilst others had potencies exceeding the upper limit, resulting in right-censoring. Although not a lot of the data was censored (approximately 20%), this is enough to be problematical, particularly as the key analysis goal was to estimate the cultivar by nutrient feed means!


Tobit Method to the Rescue!
Fortunately, censored data such as that described above can be analysed using the Tobit method, enabling us to obtain well-behaved and reliable estimates of the treatment means. In a nutshell, the Tobit method replaces the censored observations with their expected values, using an E-M (expectation-maximization) algorithm to optimize the model.
- Initialize the model parameters (i.e., of the ANOVA, LMM, GLM, GLMM or HGLM).
- E-step: Estimate the censored observations given the estimates of the model parameters.
- M-step: Maximize the likelihood, treating the estimates of the censored observations as if they were fully observed, to obtain new estimates of the model parameters.
- Iterate the E- and M- steps until convergence.
Estimating the censored observations
The censored values are estimated with their expected values. For example, for Normal data with mean m and standard deviation s:
Left-censored:
E(y | y < bound, m, s) = m − s × φ(z) / Φ(z)
where z = (bound − m) / s
Right-censored:
E(y | y > bound, m, s) = s × φ(z) / Φ(z) − m
where z = (m − bound) / s
Here, ɸ and Φ are the standard Normal PDF and CDF, respectively.
Analysis of the Censored CBD Data
We’ll analyse the censored CBD data using a Tobit linear mixed model. Here, the fixed model comprises of the main effects of nutrient feed and cultivar plus their interaction, and the random model reflects the split-plot design of replicates and rows (i.e., whole-plots) within replicates. This model is very easy to fit in Genstat, either via the menus or using code.

The analysis proceeds as follows:
- Initial estimates for the censored observations are set. By default, Genstat uses the censoring bound, but you can supply your own using the “Give initial values for censored y” field.
- The LMM is fitted using REML.
- The means (i.e., fitted values) and standard deviations of the censored observations are saved.
- The censored observations are replaced by their expected values (see box above).
- Steps 2-4 are repeated until convergence. By default, Genstat deems that the algorithm has converged when the percentage change in each estimated treatment mean is less than 0.1%, and the percentage change in each estimated variance component is less than 0.05%. However, this can be changed in the Options menu.
Genstat provides all the usual array of output you’d expect from a LMM analysis, such as significance tests for the fixed effects, estimated means, contrasts and pairwise comparisons, residual diagnostic plots, information criterion, etc. etc. etc.!
As in this example, the main goal was to estimate and compare the cultivar by nutrient feed treatment means, let’s look at a plot of the predicted means. In the following plot, the means for the new nutrient feed are plotted with blue circles, and the standard feed with yellow diamonds. An approximate least significance difference at the 5% level (~LSD(5%)) is also given.

From the means plot, we can see that the new nutrient feed has given an advantage for cultivars B and C. However, for cultivar A, which has a low overall mean CBD potency, there’s little difference between the new and standard feeds.
Censored Data Analysis with Genstat
Censored data analysis can be a challenge, but the latest Genstat release brings you new procedures and extensions specifically designed to handle censored data effectively.
Here’s what’s in Genstat for censored data analysis:
Procedures: Analyse censored data using ANOVA and censored regression analyses with Normal, gamma, and negative binomial data.
Enhanced Existing Facilities: Existing capabilities like LMM, log-Poisson GLM, GLMM, and HGLM now support left- and right-censoring, even with different bounds within the same dataset.
Menu-Driven Convenience: Access these functionalities conveniently through user-friendly menus, streamlining your workflow.
Related tutorial videos:
Linear mixed models for censored data: https://youtu.be/CTNnkxqvC-k
Censored Poisson count data: https://youtu.be/ClHluySIJ38
Take Your Analysis Further
Whether you’re working with censored concentration data, experimental yields, or count data, Genstat gives you the tools to analyse it robustly and reliably — no workarounds needed. Our consultancy team can also help you design, run, and interpret your analyses, ensuring you get the most out of your data.
Explore Genstat
Contact our consultancy team to discuss your project.