Misleading description

Statistically significant

Look out for results that are reported as “statistically significant” or “not statistically significant”.

In treatment comparisons, statistical significance is the probability of the observed difference (the effect estimate) or a bigger difference having occurred by chance if in reality there was no difference.

A “statistically significant” result is unlikely to have happened by chance. The usual threshold for this judgement is a probability of less than 5% (0.05).

However, statistical significance does not tell us anything about how important an effect is. A small, unimportant effect can be “statistically significant”. Similarly, a large, important effect can be “statistically non-significant”.

REMEMBER: Claims that results were significant or non-significant usually mean that they were statistically significant or statistically non-significant. This is not the same as important or not important. Do not be misled by such claims.