Unreliable summary

Unsystematic reviews or summaries

Look out for reviews (or summaries) of studies comparing interventions if the reviews were not carried out systematically.

There is often more than one study that has compared the same interventions. Those studies can have different results. This can happen because some studies were fair comparisons and others were not. Or it can happen because they were small studies with few subjects or events.

When the results of those studies are summarised unsystematically, it can make you think that the effects of the interventions are larger or smaller than they are. For example, a summary that pays more attention to studies that found bigger effects might make the intervention effects seem bigger than they are. This can often be the case in a traditional literature review.

Researchers protect against mistakes like these by being systematic. They start out with a plan to find, appraise and summarise all the studies that compare the same interventions, and stick to that plan. Such summaries are called systematic reviews. Even reviews that claim to be systematic may not be.

Wherever possible, use systematic reviews of fair comparisons rather than non-systematic reviews to inform decisions.

REMEMBER: Think about whether a summary of studies comparing interventions was a systematic review.