Environmental
BEWARE of claims that have an untrustworthy basis
Many claims about the effects of interventions are not trustworthy. Often this is because the reason (the basis) for the claim is not trustworthy. You should be careful when you hear claims that are:
• Too good to be true
• Based on faulty logic
• Based on trust alone
Interventions intended to benefit may also be harmful to the environment.
Most claims that an intervention always works turn out to be wrong.
We can rarely, if ever, be 100% certain about the effects of interventions.
Interventions that should work in theory often do not work in practice.
A change in environment may be associated with an action but that does not mean it is caused by the action.
More data is not necessarily better data, whatever the source.
Unless an intervention is compared to something else, it is not possible to know what would happen without it.
If a single intervention comparison (study) shows that it has a good or bad effect it does not mean that is the final answer.
Widely practiced interventions that have been used for a long time are not necessarily beneficial or safe.
Just because an intervention is new, expensive, technologically impressive, or brand-named does not mean that it is better or safer than other interventions.
Increasing the amount or intensity of an Intervention does not necessarily increase the benefits and may cause environmental harm.
Conflicting interests may result in misleading claims about the effects of interventions. Someone with an interest in getting people to use an intervention, such as making money, may overstate benefits and ignore possible harmful effects.
Personal experiences or anecdotes (stories) are an unreliable basis for assessing the environmental impacts of most actions.
Just because a claim is made by an expert or authority, you cannot be sure that it is trustworthy.
Peer-reviewed and published studies may not provide reliable estimates of effect.
THINK 'FAIR' – and check the evidence from choice comparisons
Evidence from comparisons of interventions can fool you. You should think carefully about the evidence that is used to support claims about the effects of interventions. Look out for:
• Unfair comparisons of interventions
• Unreliable summaries of comparisons
• How intervention effects are described
Look out for intervention comparisons where the comparison groups were not alike. Comparison groups need to be similar at the beginning of a comparison.
Look out for comparisons of interventions between studies that are different.
Look out for intervention comparisons where the groups were treated differently. Comparison groups should be treated equally.
Look out for treatment comparisons where people knew which treatment they received and knowing that could have changed how they felt or behaved.
Look out for intervention comparisons where what happened was measured differently in the comparison groups. Impacts should be assessed similarly.
Look out for outcomes that were not assessed reliably in intervention comparisons.
Look out for intervention comparisons where what happened was not measured in all of the original subjects. All subjects should be followed up.
Look out for intervention comparisons where subject’s outcomes were not counted in the group to which they were assigned. Subjects’ outcomes should be analysed in their original groups.
Look out for reviews or summaries of multiple studies comparing interventions that were not done systematically. Reviews of fair comparisons should be systematic.
Look out for unpublished results of fair comparisons. All results of studies should be reported otherwise estimates of effect of interventions may be biased.
Look out for intervention comparisons that are sensitive to assumptions that are made.
Look out for study results that are described as relative effects. Relative effects of interventions alone can be misleading.
Look out for intervention effects that are described as average differences. Average measures of effects can be misleading.
Look out for intervention effects that are based on small studies with few subjects. Fair comparisons with few subjects or effect measures can be misleading.
Look out for results that are reported for selected subgroups within a study or systematic review. Subgroup analyses may be misleading.
Look out for results that are reported using p-values instead of confidence intervals. Confidence intervals should be reported.
Look out for results that are reported as "statistically significant" or "not statistically significant". Don’t confuse "statistical significance" with "importance".
Look out for a "lack of evidence" being described as evidence of "no difference" in effect.
TAKE CARE – and make good choices
Good choices depend on thinking carefully about what to do. Think carefully about:
• What the problem is and what your options are
• Whether the evidence is relevant to the problem and options
• Whether the advantages outweigh the disadvantages
When you are thinking about choices for intervention, make sure that you understand what the environmental problem is and what your choices are.
Always ask yourself whether the outcomes measured in evidence sources matter to you and/or to your environmental goals.
Always ask yourself if your subjects are very different from the subjects studied.
Always ask yourself if the intervention is relevant or practical in your setting?
Always ask yourself if fair comparisons of interventions were conducted in circumstances that are relevant.
Always ask yourself whether the possible advantages of an intervention outweigh the disadvantages.
Always ask yourself how sure you are that the possible advantages of an intervention are better than the possible disadvantages.
