Faulty logic

“Lots of data!”

More data is not necessarily better data, whatever the source.

Claims based on lots of data can be misleading. Sometimes this is called “big data” (data from large databases) or “real world data” (routinely collected data). Unfortunately, routinely collected data often does not include data about “confounders”.

Confounders are factors other than the interventions being compared that can affect the outcomes. For example, a study might compare older adults who are more physically active to older adults who are less physically active to find out if being more active improves wellbeing. If the people who are more active are also more likely to spend time with others in those activities, that would be a confounder, since social relationships can also affect wellbeing.

When using routinely collected data, it is only possible to control for confounders that were already known and were measured when the data was collected. So, we cannot be sure that an association between an intervention and an outcome means that the intervention caused the outcome, rather than confounders.

BEWARE of claims that an intervention has an effect based on an association found using “big data” or “real world data”.

REMEMBER: Think about whether you can be sure that there aren’t other reasons for the association.