How to Analyse Censored Data with Genstat

A background showing bacteria with a green plant

Analysis of Censored Data

A graph with green peaks


Censoring is a common problem faced when analysing experimental data. It occurs during data collection when measurements cannot, or are not, taken above or below a bound. For example, left-censoring is a typical feature of chemical concentration data due to the lower limit of quantification. This is the lowest concentration that can be reliably measured by the measurement procedure. Concentrations that fall below this limit will therefore be left-censored. This means we don’t know the true concentration, only that it is less than the limit of quantification.


A nice example of right-censored data was recently encountered by my colleague, David, when analysing the growth of bacterial isolates on agar plates. Growth was measured by the area of the agar plate covered by the bacteria. However, some bacteria grew so well that the entire agar plate was covered at the conclusion of the experiment, resulting in right-censoring. Another example was in a wheat trial, where some grain was spilled from several bags containing the harvested plot yields, and so the yield was known to be above the recorded weight for these plots.  

Despite censoring posing a ubiquitous problem in the analysis of experimental data, how to robustly analyse censored data is a challenge! Often ad hoc solutions are resorted to, such as replacing left-censored observations with half the limit of quantification, or the censoring is ignored altogether – both of which are unsatisfactory.

To provide you with the tools you need to robustly analyse your censored data, the Genstat team have implemented a suite of specialised facilities. These include the ability to analyse censored Normal data using an ANOVA or a linear mixed model (LMM), as well as censored Poisson, gamma or negative binomial data using a generalized linear model (GLM), generalized linear mixed model (GLMM) or a hierarchical generalized linear model (HGLM). 

Example: Censored Cannabidiol Data

An experiment was conducted to compare the mean cannabidiol (CBD) potency of three cultivars nourished with two different nutrient feeds: the standard feed and a new feed. The experiment was laid-out as a split-plot, with nutrient feeds applied to the whole-plots (here, rows), and with 12 replicates.

At harvest, the percentage of CBD relative to plant wet weight (i.e., the CBD potency) was measured using an analytical procedure that had a lower limit of quantification of 2% and upper limit of 15%. Unfortunately, some plants had potencies less than the lower limit, resulting in left-censoring, whilst others had potencies exceeding the upper limit, resulting in right-censoring. Although not a lot of the data was censored (approximately 20%), this is enough to be problematical, particularly as the key analysis goal was to estimate the cultivar by nutrient feed means!

A blue and yellow table showing results from an experiment

Tobit Method to the Rescue!


Fortunately, censored data such as that described above can be analysed using the Tobit method, enabling us to obtain well-behaved and reliable estimates of the treatment means. In a nutshell, the Tobit method replaces the censored observations with their expected values, using an E-M (expectation-maximization) algorithm to optimize the model.

  1. Initialize the model parameters (i.e., of the ANOVA, LMM, GLM, GLMM or HGLM).
  2. ⇅
  3. E-step: Estimate the censored observations given the estimates of the model parameters.
  4. M-step: Maximize the likelihood, treating the estimates of the censored observations as if they were fully observed, to obtain new estimates of the model parameters.
  5. Iterate the E- and M- steps until convergence.

Estimating the censored observations

The censored values are estimated with their expected values. For example, for Normal data with mean m and standard deviation s:

Left-censored:
E(y | y < bound, m, s) = m − s × φ(z) / Φ(z)
where z = (bound − m) / s

Right-censored:
E(y | y > bound, m, s) = s × φ(z) / Φ(z) − m
where z = (m − bound) / s

Here, ɸ and Φ are the standard Normal PDF and CDF, respectively.


Analysis of the Censored CBD Data

We’ll analyse the censored CBD data using a Tobit linear mixed model. Here, the fixed model comprises of the main effects of nutrient feed and cultivar plus their interaction, and the random model reflects the split-plot design of replicates and rows (i.e., whole-plots) within replicates. This model is very easy to fit in Genstat, either via the menus or using code. 

A screenshot of the Genstat menu and specific settings

The analysis proceeds as follows:

  1. Initial estimates for the censored observations are set. By default, Genstat uses the censoring bound, but you can supply your own using the “Give initial values for censored y” field. 
  2. The LMM is fitted using REML.
  3. The means (i.e., fitted values) and standard deviations of the censored observations are saved.
  4. The censored observations are replaced by their expected values (see box above).
  5. Steps 2-4 are repeated until convergence. By default, Genstat deems that the algorithm has converged when the percentage change in each estimated treatment mean is less than 0.1%, and the percentage change in each estimated variance component is less than 0.05%. However, this can be changed in the Options menu.

Genstat provides all the usual array of output you’d expect from a LMM analysis, such as significance tests for the fixed effects, estimated means, contrasts and pairwise comparisons, residual diagnostic plots, information criterion, etc. etc. etc.! 

As in this example, the main goal was to estimate and compare the cultivar by nutrient feed treatment means, let’s look at a plot of the predicted means. In the following plot, the means for the new nutrient feed are plotted with blue circles, and the standard feed with yellow diamonds. An approximate least significance difference at the 5% level (~LSD(5%)) is also given.

A scatter graph looking at the predicted mean CBD potency and cultivars

From the means plot, we can see that the new nutrient feed has given an advantage for cultivars B and C. However, for cultivar A, which has a low overall mean CBD potency, there’s little difference between the new and standard feeds. 

Censored Data Analysis with Genstat

Censored data analysis can be a challenge, but the latest Genstat release brings you new procedures and extensions specifically designed to handle censored data effectively.

Here’s what’s in Genstat for censored data analysis:

Procedures: Analyse censored data using ANOVA and censored regression analyses with Normal, gamma, and negative binomial data.

Enhanced Existing Facilities: Existing capabilities like LMM, log-Poisson GLM, GLMM, and HGLM now support left- and right-censoring, even with different bounds within the same dataset.

Menu-Driven Convenience: Access these functionalities conveniently through user-friendly menus, streamlining your workflow.

Related tutorial videos: 

Linear mixed models for censored data: https://youtu.be/CTNnkxqvC-k

Censored Poisson count data: https://youtu.be/ClHluySIJ38

Take Your Analysis Further

Whether you’re working with censored concentration data, experimental yields, or count data, Genstat gives you the tools to analyse it robustly and reliably — no workarounds needed. Our consultancy team can also help you design, run, and interpret your analyses, ensuring you get the most out of your data.

Explore Genstat

Contact our consultancy team to discuss your project.

Picture of Dr. Vanessa Cave
Dr. Vanessa Cave
Dr. Vanessa Cave is an applied statistician interested in the application of statistics to the biosciences, in particular agriculture and ecology, and is a developer of the Genstat statistical software package. She has over 15 years of experience collaborating with scientists, using statistics to solve real-world problems. Vanessa provides expertise on experiment and survey design, data collection and management, statistical analysis, and the interpretation of statistical findings. Her interests include statistical consultancy, mixed models, multivariate methods, statistical ecology, statistical graphics and data visualisation, and the statistical challenges related to digital agriculture. Vanessa is a past President of both the Australasian Region of the International Biometric Society and the New Zealand Statistical Association, on the Editorial Board of The New Zealand Veterinary Journal and an honorary academic at the University of Auckland. She has a PhD in statistics from the University of St Andrew.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Related articles

How much of the variation in your field trial comes from the treatments, and how much comes from...
What should you do when count data are more variable than your model expects? Vanessa Cave and David...
Precision breeding can deliver scientifically impressive traits, but practical value depends on more than technical success. Owen McIntosh...