Agricultural
BEWARE of claims that have an untrustworthy basis
Many claims about the effects of interventions are not trustworthy. Often this is because the reason (the basis) for the claim is not trustworthy. You should be careful when you hear claims that are:
• Too good to be true
• Based on faulty logic
• Based on trust alone
People often think about the benefits of interventions and ignore possible harms. But few intervention are 100% safe.
Most claims that an intervention will always be 100% effective in all situations turn out to be wrong.
We can rarely, if ever, be 100% certain about the effects of interventions.
Not all interventions are necessary. Sometimes an intervention will make no difference and may even make things worse.
Interventions that should work in theory often do not work in practice.
A change in yield or agricultural output that is associated with an intervention doesn’t mean that it is necessarily that intervention that caused the change.
More data is not necessarily better data, whatever the source.
Unless an intervention is compared to something else, it is not possible to know what would happen without it.
If a single study shows that there is a good or bad effect of an intervention, it does not mean that is the final answer.
Just because an intervention is widely practiced or has been used for a long time, it does not mean that it is beneficial or safe.
Just because an intervention is new, expensive, technologically impressive, or brand-named does not mean that it is better or safer than other interventions.
Increasing the amount or intensity of an intervention does not always increase the benefits and may cause harm.
Earlier detection is not necessarily better.
It is rarely possible to know in advance which specific situations/locations will benefit or not from an intervention.
Conflicting interests may result in misleading claims about the effect of interventions. Someone with an interest in getting people to use an intervention, such as making money, may overstate benefits and ignore possible negative effects.
Personal experience or anecdotes (stories) are an unreliable basis for assessing the impacts of many actions.
Just because an intervention claim is made by an expert or authority, you cannot be sure that it is trustworthy.
“Peer-reviewed” and published studies may not be fair comparisons.
THINK 'FAIR' and check the evidence from comparisons
Evidence from comparisons of interventions can fool you. You should think carefully about the evidence that is used to support claims about the effects of interventions. Look out for:
• Unfair comparisons of different interventions
• Unreliable summaries of comparisons
• How intervention effects are described
Look out for intervention comparisons where the comparison groups were not alike.
Look out for comparisons of interventions between studies that are different.
Look out for intervention comparisons where the comparison groups were cared for differently.
Look out for intervention comparisons where people knew which intervention they received and knowing that could have changed how they felt or behaved.
Look out for intervention comparisons where what happened was measured differently in the comparison groups.
Look out for outcomes that were not assessed reliably in intervention comparisons.
Look out for intervention comparisons where what happened was not measured in all of the original subjects.
Look out for intervention comparisons where subject’s outcomes were not counted in the group to which they were assigned.
Look out for reviews (or summaries) of studies comparing interventions if the reviews were not carried out systematically.
Look out for unpublished results of fair comparisons. All results of studies should be reported (even where they are unfavourable, or the effects are minimal)
Look out for intervention comparisons that are sensitive to assumptions that are made.
Look out for intervention effects that are described just using words.
Look out for intervention effects that are described as relative effects.
Look out for intervention effects that are described as average differences.
Look out for intervention effects that are based on small studies with few subjects.
Look out for results that are reported for a selected subgroup within a study or systematic review.
Look out for results that are reported using p-values instead of confidence intervals.
Look out for results that are reported as “statistically significant” or “not statistically significant”.
Look out for a “lack of evidence” being described as evidence of “no difference”.
TAKE CARE and make good choices
Good intervention choices depend on thinking carefully about what to do. Think carefully about:
• What the problem is and what your options are
• Whether the evidence is relevant to your problem and options
• Whether the advantages outweigh the disadvantages
When you are thinking about interventions, make sure that you understand what the problem is and what your choices are.
Always ask yourself whether the intervention outcomes that are important to you have been measured in fair comparisons.
Always ask yourself if the intervention comparisons included only subjects (e.g. crops, livestock, farms) that are very different to the ones that you are interested in.
Always ask yourself if the interventions evaluated in fair comparisons are relevant.
Always ask yourself if fair comparisons of interventions were conducted in circumstances that are relevant.
Always ask yourself whether the possible advantages of an intervention outweigh the disadvantages of the intervention.
Always ask yourself how sure you are that the possible advantages of an intervention are better than the possible disadvantages of the intervention.
