Social Welfare
BEWARE of claims that have a bad basis
Many claims about the effects of treatments are not trustworthy. Often this is because the reason (the basis) for the claim is not trustworthy. You should be careful when you hear claims that are:
• Too good to be true
• Based on faulty logic
• Based on trust alone
People often think about the benefits of interventions and ignore possible harms. But few interventions that work are 100% safe.
Most claims that an intervention will make you 100% better or that it works for everyone turn out to be wrong.
We can rarely, if ever, be 100% certain about the effects of interventions.
Interventions are not always necessary- people can improve their situation or resolve their problem without an intervention. Sometimes a treatment will not help and may even make things worse.
Interventions that should work in theory often do not work in practice.
Just because using an intervention is associated with an improved or worse outcome for someone, that doesn’t mean that the intervention made them better or worse.
More data is not necessarily better data, whatever the source.
Unless an intervention is compared to something else, it is not possible to know what would happen without it.
If a single study shows that people who got one intervention did better or worse than people who got something else, it does not mean that is the final answer.
Just because something has been used for a long time or by many people, it does not mean that it helps or that it is safe.
Just because an intervention is new, expensive, technologically impressive, or brand-named does not mean that it is better or safer than other interventions.
Taking more of a treatment often increases harms without increasing how much it helps.
Someone with an interest in getting people to use an intervention or adopt a program, such as making money, may overstate benefits and ignore possible harmful effects.
If someone got better after receiving an intervention does not necessarily mean that the intervention made them better.
Just because an intervention claim is made by an expert or authority does not mean that it is trustworthy.
“Peer-reviewed” and published studies may not be fair comparisons.
THINK 'FAIR' and check the evidence from treatment comparisons
Evidence from comparisons of treatments can fool you. You should think carefully about the evidence that is used to support claims about the effects of treatments. Look out for:
• Unfair comparisons of treatments
• Uncareful summaries of comparisons
• How treatment effects are described
Look out for treatment comparisons where the comparison groups were not alike.
Look out for comparisons of interventions between studies that are different.
Look out for intervention comparisons where the comparison groups were cared for differently.
Look out for comparisons where people knew which intervention they received and knowing that could have changed how they felt or behaved.
Look out for comparisons where what happened was measured differently in the comparison groups.
Look out for outcomes that were not assessed reliably in intervention comparisons.
Look out for comparisons where what happened was not measured in lots of people or where people dropped out of the study.
Look out for summaries of studies comparing interventions that were not done systematically.
Look out for unpublished results of fair comparisons.
Look out for treatment comparisons that are sensitive to assumptions that are made.
Look out for treatment effects that are described just using words.
Look out for treatment effects that are described as average differences.
Look out for treatment effects that are based on small studies with few people.
Look out for results that are reported for a selected group of people within a study or systematic review.
Look out for results that are reported using p-values instead of confidence intervals.
Look out for results that are reported as “statistically significant” or “not statistically significant”.
Look out for a “lack of evidence” being described as evidence of “no difference”.
TAKE CARE and make good choices
Good treatment choices depend on thinking carefully about what to do. Think carefully about:
• What your problem is and what your options are
• Whether the evidence is relevant to your problem and options
• Whether the advantages are better than the disadvantages
When you are thinking about interventions, make sure that you understand what the problem is and what your choices are.
Always ask yourself whether the intervention outcomes that are important to you have been measured in fair comparisons.
Always ask yourself if the treatment comparisons included only people (or animals) that are very different from you.
Always ask yourself if the interventions evaluated in fair comparisons are relevant.
Always ask yourself if fair comparisons of interventions were conducted in circumstances that are relevant.
Always ask yourself whether the possible advantages of an intervention outweigh the disadvantages.
Always ask yourself how sure you are that the possible advantages of an intervention are better than and the possible disadvantages.
