Look out for treatment effects that are based on small studies with few people.
Sometimes there were few people in studies of the effects of treatments, or there were few people whose communication, for example, improved or worsened. The results of studies like these are sometimes described as though we can be sure about the results. However, we cannot be sure that what seems to be a difference in outcomes was because of a real difference in the effects of the treatments that were compared: it may have occurred by chance.
In other words, some people’s communication may have got better or worse anyway, regardless of the treatment they received. It is possible that there just happened to be more of those people in one group than the other. The populations we work with in speech and language therapy are also very heterogeneous- everybody is unique. This makes small sample sizes even more problematic.
It is like tossing coins. Imagine you have two coins that are exactly the same. You would expect to get the same result from tossing each coin in the air 100 times. But if you tossed each coin just 5 times, you might get “heads” 4 times with one coin and 1 time with the other coin, just by chance.
REMEMBER: You can’t be sure about treatment effects that are based on small studies with few people or few events.