Skip to content
Suppose we want to know whether having a pet dog makes people happier. There are two broad ways we could investigate this question: through an experimental study or an observational study.
In an experimental study, we actively impose different conditions and compare the resulting outcomes. For example, we might recruit a group of people and randomly assign half to have a pet dog and half not to have one, then compare their happiness after a predetermined period of time.
In an observational study, we do not impose any conditions. Rather, we simply observe what is already happening. In the pet dog example, we might survey a large group of people, record whether or not they have a dog, and compare the happiness of dog owners with that of people who do not have a dog.
At first glance, these two approaches might seem to answer the same question. But there is an important difference … people who choose to have a pet dog may differ from people who do not in many other ways. For example, they might spend more time outdoors, be more physically active, live in different types of households, or simply be the sort of people who particularly enjoy having animals around. Any of these factors might also be related to happiness. So, if people with pet dogs are happier, we cannot be sure that having the dog is actually causing the difference. Rather, the observed difference may be due to other uncontrolled variables confounding the results.
Why use an experimental study?
The major advantage of an experiment is the ability to draw stronger causal conclusions. Randomisation provides protection against confounding factors by helping to prevent systematic differences between the treatment groups (other than the treatment itself!) that might affect the response. Moreover, if we know in advance that characteristics such as age, sex or income are likely to affect the response, we can also take these into account when designing the randomisation. For example, blocking can help balance important characteristics across the treatment groups.
Experiments give us considerable control over the study. We can decide which treatments to investigate, how they are applied, when measurements are taken, and how replication, randomisation and blocking are incorporated into the design.
But experiments are not always practical. Randomly assigning people a dog is unlikely to be feasible … or particularly popular with everyone involved! Experiments can also be expensive and time-consuming, particularly when treatments need to be applied over a long period. There may also be ethical constraints. For example, we cannot randomly assign people to vape for several decades simply to investigate whether vaping causes health problems.
Experiments may also be limited to relatively modest sample sizes, as recruiting participants, applying treatments and maintaining experimental conditions can be very costly.
Why use an observational study?
Observational studies are often easier, quicker and less expensive to conduct. In many situations, the data may already exist, for example in government surveys, health records, census data or administrative databases. This can make it possible to study very large numbers of individuals and to investigate outcomes that are rare or develop over long periods.
Observational studies are also essential when experimentally imposing a treatment would be impossible or unethical.
Observational studies may also reflect what’s actually happening in the real world more closely than experimental studies. This is because, rather than imposing experimental conditions, we observe behaviour and outcomes as they naturally occur.
However, there is a crucial trade-off: the groups being compared may differ in ways other than the factor we are interested in. This potential confounding makes drawing causal conclusions from an observational study very risky.
Statistical modelling can help account for confounding factors that have been measured. For example, in our pet dog study, dog owners and non-dog owners might differ in age, income, education or other characteristics that are also related to happiness. Including these variables in the analysis can help account for these differences when assessing the association between having a dog and happiness. However, statistical modelling cannot guarantee that all important differences have been accounted for: there may always be important variables that were not measured or recorded.
Does a bigger sample solve the confounding problem?
Not necessarily.
Observational datasets can be enormous, and large sample sizes can give very precise estimates of associations and differences between groups. But precision and validity are different things. If an observational comparison is affected by confounding, collecting more observations does not automatically remove that confounding. Therefore, a very large observational study may provide a very precise estimate, but that estimate can still be biased.
Conversely, a carefully designed randomised experiment may provide much stronger evidence about a treatment effect using considerably fewer observations.
Of course, sample size still matters, but a bigger sample does not, by itself, solve the problem of confounding.
So which is better?
Neither approach is always better.
Experimental studies are usually preferred when the aim is to establish whether an intervention causes a change and when randomisation is practical and ethical.
Observational studies are invaluable when experiments are impossible, unethical or prohibitively expensive, when very large populations are required, or when the aim is to understand patterns occurring in the real world.
What can we actually conclude?
The key is to recognise what the study design allows us to say.
An observational study might provide convincing evidence that two variables are associated. A well-designed randomised experiment can go a step further, providing much stronger evidence that changing one of them causes a change in the other. This distinction is important because even the most sophisticated statistical analysis cannot overcome the limitations of the underlying study.
Whether you are planning an experiment or working with observational data, it helps to start by thinking carefully about where the data came from, what else might be influencing the results, and what you can reasonably conclude. From there, the right analysis can help you get the most useful information from your data.
