One reason we feel so strongly about the role of vaccines in public health is because of the massive amount of good data and quality studies that support them. And that’s why it’s frustrating when media, the Twittersphere, or the internet in general circulate rumors and poorly designed studies attacking vaccines.
The thing is– science is hard. Like, really hard. People study for up to 10 years to be a qualified researcher. And reading scientific literature can be a bit tricky. We can’t catch you up to the guy or gal who’s devoted an entire career to vaccinology or epidemiology, but here’s a handy cheat sheet to help you spot the “junk science” when it comes across your news feed:
1. It confuses correlation and causation.
This is a big one, and possibly a mantra you’ve heard before: correlation does not equal causation. Correlation is a statistical term that simply means the way two variables fluctuate appear to be related in some fashion. Any fashion. Maybe variable A going up happens at approximately the same rate as variable B going down. Maybe they increase together. Maybe it’s not a linear relationship (but that’s a bit more complicated). What’s most important to remember here is we absolutely cannot assume that one is causing the other. We simply don’t have enough information since all we know is how the variables are changing.
Take this example of Nicolas Cage films and drowning deaths. That plot looks pretty good right? And the correlation coefficient is a fairly solid 0.66. Could it be that Cage’s action-packed thrillers are inexplicably driving people towards backyard swimming pools? Anything’s possible. But the two almost certainly have nothing to do with each other and are, instead, a total coincidence. Often when two variables are correlated, there is actually an unknown third (and potentially fourth and fifth) variable that is affecting both of the events you’re examining.
2. Its sample size is small.
People suffer from a wide range of medical issues every day — sometimes they are caused by what you’re studying, but sometimes it’s just by chance that the participants being studied develop an issue. Out of a study sample of three, having one guy get hit by a bus would look like a significant trend. The larger the sample size, the less impact those random occurrences will have on your data.
3. The study is uncontrolled.
Not uncontrollable like your two-year-old nephew on a sugar-high, but uncontrolled as in lacking a control group. A control group provides a researcher something to which to compare results; it’s the closest way to estimate the counterfactual. Did the subjects get better over the course of the experiment because of a drug being tested, or would they have improved anyway? A control group that is similar to the experimental group in every way EXCEPT for the intervention can help you answer that question.
4. The results are not replicable.
One study alone (even a well designed, large-scale one) can’t prove anything. All it can do is contribute to the body of work already done by the scientific community. It takes several studies coming to the same conclusion to say anything with confidence — and even then we can’t be 100% certain. Science is purposefully self-correcting. Researchers rely on each other to validate their results. If no other researchers have been able to replicate a study’s findings, that’s a red flag.
On a related note, beware of those researchers who are only citing themselves. If an author says that there is “substantial evidence to support” a given link or a particular cause, check out the citations. Have several different research groups provided evidence to support the link? Or is it just one name (the author’s) that keeps popping up? If that author is the only one who seems to be providing that “substantial evidence,” it’s worth taking with a fistful of salt.
5. There’s a conflict of interest.
This is a sensitive but important point. When publishing a paper, authors must disclose the source of funding for their work as well as any other relevant conflicts of interest, such as ownership of a related private company. This does not necessarily invalidate the results of the experiment, but you should definitely be aware of any potential bias when reading results. If the author has a lot to gain from the study and the results seem glowing with no down-sides or limitations, be suspicious.
6. It’s published in a journal that’s not peer-reviewed.
Whenever possible, try to read the original journal article instead of relying on the popular press. Articles in general news media can be a great source to find out about new and interesting research, but remember they are necessarily interpreted by a reporter (in best cases by a science writer with a background in science; in the worst cases it’s a press release). While you’re reading the original article, make a note of the journal it appears in. Is it a reputable publication, like Nature, Journal of American Medical Association or the New England Journal of Medicine? Did articles have to pass a peer-review process, meaning that other experts read the manuscript, asked probing questions, pointed out any errors, and addressed limitations? This process is by no means perfect; mistakes can certainly still get through peer review and show up in reputable sources. But on the whole, a study appearing in a respected, peer-reviewed academic journal carries more weight than one published on a personal blog.
There’s another deadly threat — single-issue shill “journals” published entirely to push an agenda. This is the worst possible abuse of the scientific process. Some people, after being spurned by reputable journals, will go so far as to create their own journals to fabricate a veneer of legitimacy for their flawed ideas. These biased publications are a wolf in sheep’s clothing. Avoid them at all cost.
In the age of the internet, it’s getting harder to tell good science from bad. But if you follow this guide, and approach scientific articles with a healthy dose of skepticism, you’ll do fine.