Beware the three Vs of data – Volume, Variety and Velocity

You are no doubt aware of the increased discussion about the use of novel computational tools for plant and animal breeding over the last few years. We hear about topics such as machine learning (ML) and artificial intelligence (AI), and their respective use in so many everyday applications. In addition, we are all aware of the importance of making appropriate breeding decisions based on tangible and reliable pieces of information. The above tools, and many other statistical techniques, rely on data, which needs to be generated, collected, stored, processed and analysed. All of these activities are done well before we start drawing conclusions and making decisions from our breeding programmes. In this blog, we will discuss some important characteristics of data, focusing on the three Vs – Volume, Variety and Velocity.

Volume

You’ve probably heard of the term big data. This is a relatively trendy term and it refers to having very large volumes of data that are hard to deal with using our traditional software and basic IT infrastructure. For example, images from a drone flying over a crop every week for phenotyping; or, genomic information for our breeding population based on thousands or millions of DNA SNPs. Large volumes of data present unique challenges and some of the most common are associated with data storage, data retrieval, and data processing and analysis (even just to perform some basic filtering and pre-processing). However, statistical software and many computational algorithms are adapting to this volume by having more efficient routines and, in some cases, data sampling strategies. These in turn require adequate infrastructure in computing storage and processor speed, so small operations, such as calculating a simple mean, are possible!

Many new techniques (such as ML and AI) both require the management of large data volumes, so this accumulation is important. The more data we collect, the more insights we can gain from our breeding programmes. Therefore, we want this volume and we do need it to make the most of these tools. The solution to managing big data isn’t to decrease the size of the data set – the solution is to continue growing the dataset and concurrently improve our understanding of it. In addition, it is important that we retain and document our data so that it is available in the future for uses that we aren’t aware of today. So, we have to plan ahead to ensure we have the required infrastructure to make the most of the data we can accumulate. This touches on the long-term value of information for an organisation.

Variety

Variety refers to the diversity of the data we encounter. For example, soil microbiome data is very different from consumer preference data from surveys, although both might contribute to the same business objective. Thus, as you might expect, they have very different issues and challenges associated with them and, of course, they will require very different statistical procedures in their analysis.

Variety is associated with how diverse and complex this data is. One important characteristic of big data is its complexity. Typically, big data comprises not only large numerical data arrays but also different types of variables including discrete elements. However, this complexity often comes with additional variabilities, such as missing data, errors or inconsistencies. All of these issues present new challenges for processing and analysis, and we often require intense computational techniques (such as imputation), or the need to create new statistical or analytical tools.

On top of this variety, we need to add that the data might have different uses, hence, a variety of potential applications. Again, the solution is not to decrease your number of data sources to make your set more manageable – the solution is to invest in tools that help you consolidate multiple data sources into one comprehensive data set. As data accumulates, it might help to answer different questions and that is vital for innovation. For example, it is not uncommon to use field data to assess the long-term progress in genetic gain from a breeding programme. This variety of use implies that data has to be properly documented, so it can be used in the future. This reiterates again, the long-term value of information.

Velocity

There are a few important aspects associated with velocity. The first one is about computational speed. We can’t wait days or weeks to have our array of predictive models fitted to decide which one is the most adequate. The infrastructure associated with storage, retrieval, processing and statistical algorithms must be in place in order to analyse our data in a reasonable time frame. For example, waiting too long to obtain our genetic values will affect our selection of outstanding genotypes and this is critical for most breeding decisions. Additionally, with the emergence of precision breeding, real-time analysis of data has become even more valuable.

The other aspect is associated with the velocity and flexibility of our strategies. This refers to how quickly we can react to new challenges given the data we have at hand. This is the case, for example, of a new disease or a new consumer market that requires a specific resistance or adaptation of our varieties. Large volume and diversity of data should allow us to explore and develop new products within a reasonable time period. Here, our data must not only be accessible and documented but it should be robust enough to provide us with opportunities for a relatively fast reaction to these challenges.

Overall, our breeding programmes are influenced by the three Vs of volume, variety and velocity. While they may not be an issue today, they are likely to be part of the challenge in the future, and we need to be aware and ready to face them. As your data grows, as you add new data sources and as new global challenges arise that you need to combat, the three Vs will become a focus point.

And the final V we haven’t yet mentioned… VSNi. We offer a full range of statistical and genetics consultancy from data analysis services right through to design and implementation of a complete breeding strategy. Our team work closely with you to help you understand how the three Vs can impact your data and your ability to gain the right insights at the right time.

Our team of cross-disciplined experts will use their experience to guide you from the initial design of experiments through to decision making and everything in between. We deliver bespoke, customised solutions to help you achieve your goals more rapidly. If you’re interested in uncovering valuable insights across your breeding programmes, get in touch today.

Picture of The VSNi Team
The VSNi Team

Comments

One Response

Leave a Reply

Your email address will not be published. Required fields are marked *

Related articles

How much of the variation in your field trial comes from the treatments, and how much comes from...
What should you do when count data are more variable than your model expects? Vanessa Cave and David...
Precision breeding can deliver scientifically impressive traits, but practical value depends on more than technical success. Owen McIntosh...