Before you start
Most of us were taught what the literature concludes, and far fewer of us were taught how to check whether a conclusion is any good. Appraisal is a separate skill from clinical knowledge, it can be learned in an afternoon, and it travels with you into every part of practice.
You do not need statistics in order to do this, but you do need to know which questions to ask and in what order to ask them. Six questions will get you most of the way, and all six can be answered from the abstract and the methods without reading the discussion at all.
One habit is worth adopting before any of the six questions, which is to read the methods before the abstract. The abstract is where the authors tell you what they would like you to conclude, whereas the methods are where they tell you what they actually did.
The six questions
Who was in it, and are they like my patient?
Start with the species, the breed, the age, and how sick the animals were when they were enrolled. A trial conducted in young purpose-bred beagles tells you very little about a fourteen-year-old cat with kidney disease. Check how many animals were in each group rather than only the total, because a study of sixty cats split three ways is really three studies of twenty.
Warning signThe numbers are given only as a total, or the entry criteria are so narrow that almost nobody in your waiting room would have qualified.
What was it compared with?
If there was no control group then there is no comparison, and any animal that improved might have improved anyway. Look for what the control animals actually received. A placebo comparison is the strongest, a comparison against standard treatment is useful, and a comparison against nothing at all, or against a group assembled afterwards from records, is considerably weaker than either.
Warning signThe paper reports that cats improved after treatment without stating what they improved relative to.
Was the outcome something the animal cares about?
Survival, freedom from clinical signs and avoiding a hospital admission are outcomes that matter to the patient. A number on an echocardiogram, a blood value or a score on a scale are stand-ins for those outcomes, and a stand-in can move without the animal being any better off. Drugs are licensed on stand-ins regularly, which is why the distinction is worth holding onto.
Warning signThe paper measures wall thickness or a biomarker, and the discussion then talks about survival.
Did they decide what counted as success before they looked?
This is the question most readers skip, and skipping it does more damage than anything else on this list. If the outcome was chosen after the data came in, or if a subgroup appeared that had not been planned, then the result is a hypothesis rather than a finding. Look in the methods for the word prespecified, and look in the results for any subgroup that the methods never mentioned.
Warning signThe headline result lives in a single subgroup, and that subgroup is described for the first time in the results section.
How big was the effect, and could they even measure it?
A statistically significant result is not necessarily a result large enough to matter. Find the actual difference between the groups, expressed with its units, and then ask whether the equipment used could reliably detect a difference of that size. Repeated echocardiographic measurements on the same animal vary by a few per cent, so an effect of a tenth of a millimetre on a wall seven millimetres thick sits inside the measurement noise.
Warning signThe abstract carries a p-value and no effect size appears anywhere, or the effect reported is smaller than the known repeatability of the test.
Who paid, and who did the analysis?
Industry funding does not make a study wrong, and plenty of good trials are company-funded because somebody has to pay for them. What it does mean is that the design deserves closer reading, particularly the choice of comparator and the choice of endpoint. Check whether the sponsor’s own staff performed the statistical analysis, and whether the same small group of investigators appears on every paper written about the drug.
Warning signThere is no funding statement at all, or the conflicts section declares none on a paper about a product that two of the authors consult for.
Words that should slow you down
The terms below turn up constantly, and each of them changes how much weight a result will carry.
| Term | What it means for you |
|---|---|
| Surrogate endpoint | A measurement standing in for something that matters. Wall thickness instead of heart failure. Useful for deciding what to study next, weak for deciding what to prescribe. |
| Composite endpoint | Several outcomes bundled into one count, so a hospital visit and a death score the same. A positive composite is often driven by the least serious part of it. |
| Non-inferiority | The study set out to show the new treatment is not much worse, not that it is better. The margin for "not much worse" is chosen by the investigators. Always look at what it was. |
| Post hoc | Decided after seeing the data. A post hoc subgroup, endpoint or analysis is a suggestion for the next study, not a result. |
| Per protocol | Animals that dropped out were excluded from the analysis. If the sickest ones dropped out, the treatment looks better than it was. Intention to treat keeps everyone in and is the honest version. |
| Confounding by indication | The treated animals were different from the untreated ones before anything was given, usually sicker. Very common in retrospective work, and it can point a result in either direction. |
| Underpowered | Too few animals to detect a difference that would matter. A negative result from a small study means nobody looked hard enough, not that there is no effect. |
| Alpha | The threshold for calling something significant, conventionally 0.05. If a paper sets it higher, or makes several comparisons without adjusting, false positives become likely. |
Scroll sideways for the full table.
Turning the answers into a mark
Once you have been through the six questions you will have a sense of how much the paper can carry. Resist the urge to turn that into a single number based on what kind of study it was. A design ladder, on which a randomised trial always outranks a cohort study, breaks in a way that matters, because a small industry-funded trial measuring a stand-in outcome would come out ahead of a large and carefully conducted observational study. That ordering is wrong, and the grading systems used in human medicine avoid it by separating two questions rather than one. This site does the same.
Start from the design, then move down from it. A randomised trial starts high. Rate the certainty down for whatever your six questions turned up, which will usually be risk of bias, too few animals, a population unlike your patient, or results that disagree between studies. Rate it up for a large effect that appears consistently across different groups. This is how GRADE works, and it is the reason a downgraded trial and a case series can end up in the same place.
One hard rule about stand-in outcomes. If the only evidence measures wall thickness, ectopic counts or a blood value rather than something the animal experiences, it cannot rise above the bottom band, whatever the design. This cap is an editorial rule adopted here rather than a part of GRADE. The principle behind it comes from SORT, the grading scheme used in human family medicine, which gives its weakest recommendation grade to evidence resting on disease-oriented outcomes; SORT grades recommendations rather than bodies of evidence, so the hard cap is a choice made here. It exists because improvements in a measurement are not reliably followed by improvements in the patient.
Record what the evidence points to as a separate mark. Certainty and direction are different questions, and you can hold low certainty that something helps, or reasonable certainty that it does nothing at all. Keeping the two apart prevents "we do not know" from being read as "it does not work", and prevents "no evidence of benefit" from being read as "evidence of no benefit". These two pairs are confused constantly, and in both directions.
The ACVIM guidelines on mitral valve disease already define their weakest evidence category as including randomised studies that carry limitations of design or execution, so the specialty has published the same point.
Practise on something
Reading about appraisal achieves less than working through one paper yourself. Two worked examples on this site, both from feline cardiology, are suitable for that purpose.
-
Rapamycin for subclinical HCM
This is a licensed drug for which the endpoint was chosen after the data came in, and for which the reported effect is smaller than the measurement error of the test used to find it. It is good practice for questions four and five.
-
Steroids and heart failure in cats
This is a belief taught almost everywhere, resting on two papers that do not support it. It is good practice for question two, and for recognising when a temporal association is being asked to do work it cannot do.
Work through the six questions on each paper before reading the commentary, and then compare where your reading and the commentary agree.
Checking that a reference is real
Checking a reference is quick and it is worth doing. References are miscopied between papers for years at a time, and software that writes text for you will produce citations that look entirely plausible and do not exist.
-
Search the title, not the authors
Paste the exact title into PubMed, and if nothing comes back try Crossref and the journal's own archive, since a good deal of the veterinary literature is not indexed in PubMed at all. Author names get misspelled and initials get dropped, but a title either matches or it does not.
-
Resolve the DOI
Put doi.org/ in front of the identifier, and it should land on the paper itself. A DOI that fails to resolve is worth a second look, but it is not proof of invention on its own: newly registered DOIs can take time to activate, and links get mistyped or truncated. Check the title in PubMed, Crossref or the journal's own archive before concluding anything. Some journals do not issue DOIs at all, so a missing one means nothing either way.
-
Check the volume, year and pages against the record
Volume and page numbers are where genuine errors hide. A real paper carrying the wrong page range has usually been copied from another paper's reference list rather than read.
The whole thing on one card
- Who was in it, and are they like my patient?
- What was it compared with?
- Is the outcome something the animal cares about?
- Was success defined before they looked at the data?
- How big is the effect, and can the test measure it?
- Who paid, and who ran the numbers?
This page prints cleanly if you want it on paper for a journal club, using your browser's print command.
Still to come
Worked appraisals of common presentations are planned, together with a short glossary of the statistics that recur in veterinary trials and a set of discussion prompts for journal clubs. If there is a paper you would like taken apart here, write to contact@evidencegap.vet.