Fitting mixed models in Python with ASReml-R

NOTE: This is a short tutorial, but before we start, please check out the following pre-requisites to succesfully run the code. You will need:

  1. Python installed. If you do not have it, find some online tutorials to guide you on how to install it.
  2. R with ASReml-R installed and licensed. In case you don’t have it, full instructions to install and purchase it are available at: https://vsni.co.uk/software/asreml-r.
  3. Jupyter notebook instaled. Refer to the official website for instructions on how to install it: https://jupyter.org.
  4. Internet access. For some of the Python packages and to check the ASReml-R license.

Foreword

Python is the most popular programming language for data science and related fields. It is versatile, user friendly, well documented and is backed by a vast and ever increasing community. In addition, its libraries often have more permissive terms of licensing, and this generates significant interest, specially from industry.

If you are a Python user you might have noticed that despite the myriad of avaiable libraries and frameworks – specially on the Machine Learning side – this language still lacks some conventional, well-stablished and proven statistical tools, such as linear mixed models (LMM).

LMMs has been around for decades, and they are the reference statistical method to analyze unbalanced, heterogeneous, correlated, and complex experimental data. This makes it a critical tool for applications in agriculture, ecology, food science, and genomics, to name a few fields.

In R we have some great libraries that can fit LMMs (and many other statistical models) that are missing in Python. Among the R libraries that can fit complex mixed models is ASReml-R (or simply, asreml). This package, and its stand-alone counterpart ASReml-SA, have been a popular choice for decades due to flexibility, efficiency, and reliability.

The purpose of this tutorial is to show you how to exploit the available tools in R, and we will use this R library ASReml-R to demonstrate how to navigate effortlessly between those two languages.

To begin, we will present two ways to facilitate this interaction:

  1. An R-focus aproach, where we explicitly call ASReml-R with R code in between Python code sections; and
  2. A Python-focus approach (but a bit more tricky), where we import and load ASReml-R as an internal Python library.

Note that these two are not the only ways to use R and Python together, and we want to just show some possibilities and caveats, so you can get started. Now, fire up a jupyter notebook and follow the steps.

Installing rpy2

In both strategies we are going to use Python’s rpy2 library as the interface between Python and R. Run the following code to install it in case you haven’t already.

!pip install --quiet pandas rpy2

Approach 1: calling ASReml-R with R code in between Python code sections

In order to get going with this approach first we need to load the packages: pandas and rpy2.

  • pandas is a library to manipulate data frames, and
  • rpy2 is an interface library to R embedded in a Python process.
import pandas as pd
import rpy2

We activate the R interface that will help us interact with R code.

%load_ext rpy2.ipython

Now we are ready! In this example we will load the oats dataset into Python directly from a file and then we have a look at it. Note that below we are only using Python code.

oats = pd.read_csv('https://downloads.vsni.digital/a984bcc71ddb47586c9cb67b49d7e3e5c4a820b3/oats.dat', sep = '\t')
print(oats)
    Blocks Nitrogen  Subplots      Variety  Wplots  yield  Column  Row  nrate
0        1  0.6_cwt         1   Marvellous       1    156       1    1    0.6
1        1  0.4_cwt         2   Marvellous       1    118       2    1    0.4
2        1  0.2_cwt         3   Marvellous       1    140       1    2    0.2
3        1    0_cwt         4   Marvellous       1    105       2    2    0.0
4        1    0_cwt         1      Victory       2    111       1    3    0.0
..     ...      ...       ...          ...     ...    ...     ...  ...    ...
67       6  0.2_cwt         4  Golden_rain       2     82       4   16    0.2
68       6    0_cwt         1   Marvellous       3     97       3   17    0.0
69       6  0.2_cwt         2   Marvellous       3     99       4   17    0.2
70       6  0.4_cwt         3   Marvellous       3    119       3   18    0.4
71       6  0.6_cwt         4   Marvellous       3    121       4   18    0.6

[72 rows x 9 columns]

To declare a notebook cell that uses R directly, we need to add the %%R command. Let’s try this by first loading the library ASReml-R that we will use for fitting our linear mixed models.

%%R
library(asreml)
R[write to console]: Carregando pacotes exigidos: Matrix



Online License checked out Tue Jun 20 13:15:24 2023


R[write to console]: Loading ASReml-R version 4.2

Before we fit our models, we need to transfer the data (in this case oats) from Python to R, so we can make it available for the library ASReml-R.

This is done by using the -i code after the %%R commnad, followed by the name of the data to be transferred. After this line is executed, we can perform any desired manipulations as we do with R, such as defining the class of variables. This is shown in the code below.

%%R -i oats
oats$Variety <- as.factor(oats$Variety)
oats$Nitrogen <- factor(oats$Nitrogen, levels = c('0_cwt', '0.2_cwt', '0.4_cwt', '0.6_cwt'))
oats$Blocks <- as.factor(oats$Blocks)
oats$Wplots <- as.factor(oats$Wplots)

Before we start our modelling within ASReml-R, we need to define our statistical model. For the oats dataset, we have a split-plot design with blocks with two main factors Variety (in whole-plots) and Nitrogen (in sub-plots), and its linear model is:

𝑦𝑖𝑒𝑙𝑑=𝑉𝑎𝑟𝑖𝑒𝑡𝑦+𝑁𝑖𝑡𝑟𝑜𝑔𝑒𝑛+𝑉𝑎𝑟𝑖𝑒𝑡𝑦:𝑁𝑖𝑡𝑟𝑜𝑔𝑒𝑛+𝐵𝑙𝑜𝑐𝑘𝑠+𝐵𝑙𝑜𝑐𝑘𝑠:𝑊𝑝𝑙𝑜𝑡𝑠+𝑒

where:

  • 𝑦𝑖𝑒𝑙𝑑 is the response variable;
  • 𝑉𝑎𝑟𝑖𝑒𝑡𝑦, 𝑁𝑖𝑡𝑟𝑜𝑔𝑒𝑛, and 𝑉𝑎𝑟𝑖𝑒𝑡𝑦:𝑁𝑖𝑡𝑟𝑜𝑔𝑒𝑛 are the main fixed effect factors and their interaction;
  • 𝐵𝑙𝑜𝑐𝑘𝑠 and 𝐵𝑙𝑜𝑐𝑘𝑠:𝑊𝑝𝑙𝑜𝑡𝑠 are random blocking terms; and
  • 𝑒 is the random residual term.

The above model can be fitted with ASReml-R as follows:

%%R
oats_asr <- asreml(
    fixed = yield ~ Variety + Nitrogen + Variety:Nitrogen,
    random = ~Blocks + Blocks:Wplots,
    residual = ~idv(units),
    data = oats)
ASReml Version 4.2 20/06/2023 13:15:25
Model fitted using the sigma parameterization.
          LogLik        Sigma2     DF     wall
 1     -271.9031           1.0     60   13:15:25    
 2     -251.1383           1.0     60   13:15:25    
 3     -229.2302           1.0     60   13:15:25    
 4     -215.7286           1.0     60   13:15:25    
 5     -210.1844           1.0     60   13:15:25    
 6     -209.4057           1.0     60   13:15:25    
 7     -209.3780           1.0     60   13:15:25    
 8     -209.3779           1.0     60   13:15:25  

We might as well ask for additional iterations over the previous model using the function update.asreml. Note that the fitted model object was called oats_asr.

%%R
oats_asr <- update.asreml(oats_asr)
ASReml Version 4.2 20/06/2023 13:15:25
Model fitted using the sigma parameterization.
          LogLik        Sigma2     DF     wall
 1     -209.3779           1.0     60   13:15:25    
 2     -209.3779           1.0     60   13:15:25    

Before, we look into the estimates of the mode, we should check the model assumptions by exploring diagnostic plots.

%%R
plot(oats_asr)

These seem reasonable, and now we can proceed to request the estimated variance components.

%%R
summary(oats_asr)$varcomp
              component std.error  z.ratio bound %ch
Blocks         214.4771 168.83405 1.270343     P   0
Blocks:Wplots  106.0618  67.87553 1.562593     P   0
units!units    177.0833  37.33244 4.743416     P   0
units!R          1.0000        NA       NA     F   0

Also, we can invoke the wald.asreml function to get an ANOVA-like table to assess the significance of our main and interaction fixed effects. The comand below will provide the correct degrees of freedom associated with this experiment (i.e., split-plot), and the approximated F-tests are conditional with respect to all other terms in the model.

%%R
wald.asreml(oats_asr, denDF = 'numeric', ssType = 'conditional')
ASReml Version 4.2 20/06/2023 13:15:28
Model fitted using the sigma parameterization.
          LogLik        Sigma2     DF     wall
 1     -209.3779           1.0     60   13:15:28    
 2     -209.3779           1.0     60   13:15:28    
$Wald

Wald tests for fixed effects.
Response: yield

                 Df denDF   F.inc   F.con Margin      Pr
(Intercept)       1     5 245.100 245.100        0.00002
Variety           2    10   1.485   1.485      A 0.27239
Nitrogen          3    45  37.690  37.690      A 0.00000
Variety:Nitrogen  6    45   0.303   0.303      B 0.93220

$stratumVariances
              df  Variance Blocks Blocks:Wplots units!units
Blocks         5 3175.0556     12             4           1
Blocks:Wplots 10  601.3306      0             4           1
units!units   45  177.0833      0             0           1

We can also use any other functions included in ASReml-R. Probably, one of the most interesting is the predict.asreml function that provides us with ‘estimated means’ of each fixed effect term of interest. In the code below, we are requesting these means for each level of Nitrogen.

Note that we are using the -o option to export the object preds from R into Python, and this will be defined as a pandas data frame.

%%R -o preds
preds <- predict(oats_asr, classify = 'Nitrogen')$pvals
ASReml Version 4.2 20/06/2023 13:15:28
Model fitted using the sigma parameterization.
          LogLik        Sigma2     DF     wall
 1     -209.3779           1.0     60   13:15:29    
 2     -209.3779           1.0     60   13:15:29    
 3     -209.3779           1.0     60   13:15:29    

And now preds is in Python! And it can be treated as other pandas object for further manipulations or exploring.

print(preds)
   Nitrogen  predicted.value  std.error     status
1         1        79.388889    7.17471  Estimable
2         2        98.888889    7.17471  Estimable
3         3       114.222222    7.17471  Estimable
4         4       123.388889    7.17471  Estimable

For this approach it was clear that Python and R are kept mostly separated in terms of environments and coding. However, the transfer from one to the other (both directions) is easily done with the rpy2 library. Hence, the capabilities of ASReml-R or any other R functions and/or libraries can be safely implemented along our Python code.

Approach 2: import and load ASReml-R as an internal Python library

This approach might seem be a bit more familiar for Python users, but there are a few caveats to keep in mind. For example, the usage of dots (.) in functions and object names is very common in R, but in Python these are conflictive; hence, for compatibility, all dots need to be replaced by underscores (_). For illustration, instead of using summary.asreml we will have to use summary_asreml. Nevertheless, one important advantage is that we now can ignore all of the R code, and use just plain Python code.

In order to get going with this approach first we need to import some requires packages.

  • rpy2 and pandas libraries (as done in the previous approach),
  • importr function used to import R packages,
  • Formula function (as f for convenience) to write R formulas from strings, and
  • pandas2ri function to enable automated conversion between pandas objects to R object. Note that this has to be activated.

The code required for the above elements is shown next.

import rpy2
import pandas as pd
from rpy2.robjects.packages import importr
from rpy2.robjects import Formula as f
from rpy2.robjects import pandas2ri
pandas2ri.activate()

We proceed to import the R library ASReml-R (names as asreml) using the importr function.

asreml = importr('asreml')

Now, ASReml-R can be used as if it was any Python library, and its functions can be accessed via the code asreml.<function name> (just as for R using asreml::<function name>).

As an illustration, let’s import the oats dataset and do some testing.

oats =
pd.read_csv('https://downloads.vsni.digital/a984bcc71ddb47586c9cb67b49d7e3e5c4a820b3/oats.dat', sep = '\t')

For Python, if some of the variables have to be transformed into categories (“factors” in R lingo), this can be done using the astype() and CategoricalDtype() functions. This is done below, and we also use oats.dtypes to check the type of each column.

oats['Variety'] = oats['Variety'].astype('category')

oats['Nitrogen'] = oats['Nitrogen'].astype(
    pd.CategoricalDtype(categories = ['0_cwt', '0.2_cwt', '0.4_cwt', '0.6_cwt'], ordered = True))

oats['Blocks'] = 'B' + oats['Blocks'].astype(str)
oats['Blocks'] = oats['Blocks'].astype(
    pd.CategoricalDtype(categories = ['B1', 'B2', 'B3', 'B4', 'B5', 'B6'], ordered = True))

oats['Wplots'] = 'W' + oats['Wplots'].astype(str)
oats['Wplots'] = oats['Wplots'].astype(
    pd.CategoricalDtype(categories = ['W1', 'W2', 'W3'], ordered = True))

print(oats.dtypes)
Blocks      category
Nitrogen    category
Subplots       int64
Variety     category
Wplots      category
yield          int64
Column         int64
Row            int64
nrate        float64
dtype: object

The linear mixed model that was defined for approach 1, based on a split-plot design with blocks, can now be fitted with ASReml-R as shown below. Note that all results are sent to the object oats_asr.

oats_asr = asreml.asreml(
    fixed = f('yield ~ Variety + Nitrogen + Variety:Nitrogen'),
    random = f('~Blocks + Blocks:Wplots'),
    residual = f('~idv(units)'),
    data = oats)
ASReml Version 4.2 20/06/2023 13:15:30
Model fitted using the sigma parameterization.
          LogLik        Sigma2     DF     wall
 1     -271.9031           1.0     60   13:15:30    
 2     -251.1383           1.0     60   13:15:30    
 3     -229.2302           1.0     60   13:15:30    
 4     -215.7286           1.0     60   13:15:30    
 5     -210.1844           1.0     60   13:15:30    
 6     -209.4057           1.0     60   13:15:30    
 7     -209.3780           1.0     60   13:15:30    
 8     -209.3779           1.0     60   13:15:30    

There are a few comments related to this code cell.

  • The asreml.asreml code calls the asreml function (second word) from the asreml package (first word).
  • The Python function f is being used to transform the strings into R formulas. This is required as Python does not have an equivalent class.
  • The oats data set is being automatically converted into an object of a class that ASReml-R can use.

We can proceedd now to check the outputs by having a look at the oats_asr object. For example, below we print the estimated variance components.

summ = asreml.summary_asreml(oats_asr)[4]
print(summ)
              component std.error  z.ratio bound %ch
Blocks         214.4771 168.83211 1.270357     P   0
Blocks:Wplots  106.0618  67.87552 1.562593     P   0
units!units    177.0833  37.33222 4.743445     P   0
units!R          1.0000        NA       NA     F   0

The use of rpy2, rx and rx2 as proxies for R’s [] and [[]] operators, respectively. Hence, to get the variance components from the fit model we can run:

summ = asreml.summary_asreml(oats_asr)
print(summ.rx2('varcomp'))
              component std.error  z.ratio bound %ch
Blocks         214.4771 168.83211 1.270357     P   0
Blocks:Wplots  106.0618  67.87552 1.562593     P   0
units!units    177.0833  37.33222 4.743445     P   0
units!R          1.0000        NA       NA     F   0

We hope this short tutorial gave you enough details on how to use ASReml-R alongside Python. It is important to note that ASReml-R was created, optimized, tested, and is supported for R, which means that some specific bits of code might require additioanl tricks. And please, let us know of your experience and needs with ASReml-R in this R/Python interface a https://vsni.co.uk/contact

Appendix

See below the session information for reference.

%%R
sessionInfo()
R version 4.2.0 (2022-04-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Monterey 12.6.3

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] C/UTF-8/C/C/C/C

attached base packages:
[1] tools     stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
[1] asreml_4.2.0.258 Matrix_1.4-1    

loaded via a namespace (and not attached):
 [1] magrittr_2.0.3    tidyselect_1.2.0  munsell_0.5.0     colorspace_2.0-3 
 [5] lattice_0.20-45   R6_2.5.1          rlang_1.0.6       fansi_1.0.3      
 [9] dplyr_1.1.0       grid_4.2.0        data.table_1.14.2 gtable_0.3.0     
[13] utf8_1.2.2        cli_3.6.0         ellipsis_0.3.2    digest_0.6.29    
[17] tibble_3.1.7      lifecycle_1.0.3   crayon_1.5.1      farver_2.1.0     
[21] ggplot2_3.3.6     vctrs_0.5.2       glue_1.6.2        labeling_0.4.2   
[25] compiler_4.2.0    pillar_1.7.0      generics_0.1.3    scales_1.2.0     
[29] pkgconfig_2.0.3  
import session_info
session_info.show(excludes = ['asreml'], html = False)
-----
pandas              1.5.3
rpy2                3.5.10
session_info        1.0.0
-----
IPython             8.3.0
jupyter_client      8.0.3
jupyter_core        5.3.0
notebook            6.5.3
-----
Python 3.9.16 (main, Dec  7 2022, 10:15:43) [Clang 14.0.0 (clang-1400.0.29.202)]
macOS-12.6.3-x86_64-i386-64bit
-----
Session information updated at 2023-06-20 13:15

Picture of Dr. Giovanni Galli
Dr. Giovanni Galli
Dr. Giovanni Galli is an Agronomist with an M.Sc. and Ph.D. in Genetics and Plant Breeding from the University of São Paulo (USP)/Luiz de Queiroz College of Agriculture (ESALQ). He currently works as a Statistical Consultant at VSN International, United Kingdom. Dr. Galli has experience in field trials, quantitative genetics, conventional and molecular breeding (genomic prediction and GWAS), machine learning, and high-throughput phenotyping.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Related articles

How much of the variation in your field trial comes from the treatments, and how much comes from...
What should you do when count data are more variable than your model expects? Vanessa Cave and David...
Precision breeding can deliver scientifically impressive traits, but practical value depends on more than technical success. Owen McIntosh...