Misleading description

Statistical significance

“Statistical significance” is often confused with “importance”. The cut-off for considering a result as statistically significant is arbitrary, and statistically non-significant results can be either informative (showing that it is very unlikely that alternative interventions differ importantly) or inconclusive (showing that the relative effects of the interventions compared are uncertain).

Claims that results were significant or non-significant usually mean that they were statistically significant or statistically non-significant. This is not the same as important or not important.

Even if the difference between two interventions is not statistically significant, it is still possible that a practically relevant difference exists. It might be that the difference is not visible because the study did not have enough participants or data points.

For example, a difference of 7 points on a Maths test may be practically relevant because it may mean that it puts students in a higher grade band. However, the study may show that there is no statistically significant difference between the group and that the mean difference is, for example, 4.3 points. However, if the practically relevant difference falls within the confidence interval (e.g. 95% CI: -1.03 to 8.4), we cannot be certain that there is not practically relevant difference between the interventions.

BEWARE not to be misled by claims that confound statistical significance with importance.

REMEMBER that statistically non-significant results can be informative too.