Unreliable summary

Unsystematic summary

Look out for summaries of studies comparing interventions that were not done systematically.

There is often more than one study that has compared the same interventions. Those studies can have different results. This can happen because some studies were fair comparisons and others were not. Or it can happen because they were small studies with few people or events.

When the results of those studies are summarised unsystematically, it can make you think that the effects of the treatments are larger or smaller than they are. For example, a summary that pays more attention to studies that found bigger effects might make the treatment effects seem bigger than they are.

Researchers protect against mistakes like these by being systematic. They start out with a plan for finding and summarising all the studies that compare the same treatments, and stick to that plan. Such summaries are called systematic reviews.

REMEMBER: Think about whether a summary of studies comparing treatments was a systematic review.