Unfair comparison

Indirect comparisons

There are many different teaching methods and possible interventions but they are rarely compared to each other in the same studies.

Usually, an intervention is compared to the ‘status quo’, the teaching method that is typically used in a given context. For example, one study may look at the effect of a Maths Mastery approach and compare this new approach to a more traditional way of teaching. Another study may look at the effect of inquiry-based teaching on maths performance and compare this approach to more traditional maths teaching.

People who want to know which approach to adopt might indirectly compare the two teaching methods by looking at how they compared to more traditional approaches.

However, there can be important difference between studies when interventions are indirectly compared. For example, the socio-economic background of students in the two studies may vary, there may be different levels of students with EAL, or students may simply differ in age across studies. These are only some of the factors that can potentially influence student outcomes and need to be taken into account when comparing results across studies.

BEWARE of claims that are based on indirect comparisons.

REMEMBER: When indirect comparisons are needed to inform intervention choices, think about whether careful consideration was given to differences between studies.