Unreliable summary

Uncareful summary

Look out for summaries of studies comparing treatments that were not done carefully.

There is often more than one study that has compared the same treatments. Those studies can have different results. This can happen because some studies were fair comparisons and others were not.
Or it can happen because they were small studies with few people or events.

When the results of those studies are summarised uncarefully, it can make you think that the effects of the treatments are larger or smaller than they are. For example, a summary that pays more attention to studies that found bigger effects might make the treatment effects seem bigger than they are.

Researchers protect against mistakes like these by being careful. They start out with a plan for finding and summarising all the studies that compare the same treatments, and the stick to that plan.

REMEMBER: Ask if a summary of studies comparing treatments was a careful summary.