Unfair comparison

Original groups

Learners’ outcomes should be counted in the group to which they were originally allocated.

Randomised allocation helps to ensure that the comparison groups have similar characteristics. However, learners sometimes do not receive the allocated intervention. The characteristics of these learners often differ from those who do receive the allocated intervention. Excluding learners who did not receive the allocated intervention from the analysis may mean that like is no longer being compared with like. This may lead to an under – or overestimate of effect relative to what would have happened if everyone had received what was intended.

For example, in a comparison of a whole-class literacy approach, classes who stop using the intervention during the evaluation should be counted in the intervention group, even though they did not receive the intervention as intended.

BEWARE of relying on the results of intervention comparisons if learners’ outcomes are not counted in the group to which they were allocated.

REMEMBER to check that learners’ outcomes were counted in their original groups.