Is Most Science News Bullshit?
Our reverence for science has led to a culture of “new findings” and sensationalistic headlines
By Simon Oxenham
If we want to live our lives based on evidence (and who doesn’t?), it makes intuitive sense to live according to what we read in the science section of the newspaper. First things first, don’t do that. Newspapers are actually one of the worst places to get information about scientific matters. Now this isn’t some kind of revelation. Anyone with a good understanding of science knows this is as clear as night and day.
Let’s begin by looking at the most widely-read news website — Mail Online, which provides a perfect demonstration of what we’ll call the seesaw effect. Almost every week, the Mail publishes news stories illustrating scientific findings that — apparently — turn our understanding of the world upside down. If you believed everything you read in the Mail about cancer, for example, you’d have to believe that everything from taking aspirin to drinking beer both causes and prevents cancer. That’s according to The Daily Mail Ontological Oncology project, a tongue-in-cheek attempt to track “the Daily Mail’s ongoing effort to classify every inanimate object into those that cause cancer and those that prevent it.”
But it’s not just the Mail that’s at fault. This version of science is endemic throughout the media. A week doesn’t go by without some news sites confidently stating that scientists have “changed their minds” in one direction or the other, whether it’s on the health consequences of wine, meat, or chocolate.
Arguably the field for which this is most true is the study of nutrition. As an editorial in the British Medical Journal once pointed out: “almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome.” While it is true that there is evidence that under certain circumstances, certain substances in certain quantities may cause or prevent illness, this doesn’t tend to be how the facts are presented in the news. This is because a well-considered and balanced article is not only time consuming and challenging to write, but ultimately sells less copies, brings in less traffic, and consequently less advertising dollars than shock-horror headlines. All journalists who want to tell substantive, well-researched stories face this problem.
In reality, science doesn’t swing back and forth like a weather vane blowing in the wind, as much of the media would have you believe. The Copernican revolution might be the citation of choice for global warming deniers and flat earthers, but complete turns in scientific understanding are in fact extremely rare.
Scientific progress has far more in common with a cruise ship, only changing its course ever so slowly, as papers appear and are either replicated or challenged again, and again, and again. Meta-analyses — evaluations of all the published research on a particular question just don’t tend to get the same level of coverage as cutting edge findings.
Now this isn’t to say that revolutions in scientific understanding don’t happen. Our understanding of the health risks of different kinds of fat are a prime example. “My mother was devastated,” a colleague told me, upon reading news stories explaining that saturated fat was not a terrible evil after all. People across America who had been eating foods that taste like plastic for decades likely felt the same way — like they had all been duped.
But my colleague’s mom shouldn’t feel silly for all those years of hard work maintaining a healthy diet. Too much saturated fat is still bad for you, it’s just not quite as bad for you as previously believed. Unfortunately, that doesn’t make quite as sexy a headline, and it sure as hell doesn’t sell magazines. Rather than offering a calm and considered analysis (kudos to Vox for doing an outstanding job examining the evidence on this subject), most news sites went into overdrive claiming that all of a sudden “Saturated Fat Is Good for You.” The truth is, saturated fats are often less fattening than the trans fats, carbohydrates, and sugars which are combined in processed foods and sold as “fat free options,” but this is a slightly more complex message that got lost in the hype.
Now before you set out with your pitchfork to hunt down your local science journalist, let’s be clear about one thing. It is not entirely fair to blame lazy or sensationalistic journalists for the problem of constantly see-sawing scientific claims in the news. The poor coverage for the saturated fat story arguably stemmed from an incomplete press release. This isn’t an isolated case. A recent analysis of science press releases found that randomized controlled trials were less than half as likely to result in a press release than observational trials, which are far more likely to produce incorrect conclusions. When randomized controlled trials are conducted, much of the time the widely publicized conclusions that were based on observational trials can be tossed straight out of the window. Another 2014 study found that more than a third of university-issued press releases were guilty of exaggeration.
The bad press release to bad journalism pipeline was elegantly exposed last year by journalist John Bohannon. He produced an obviously flawed paper claiming chocolate helps you lose weight, which was covered by news outlets as various as The Huffington Post, the Mail, and Texas television news. What’s worse is that many of the stories remain online misinforming people forever.
But “what’s the harm in a little hyperbole? If it will get people to listen,” I hear the editors of tabloids cry. The harms are many, but before we look at them we need to understand one very important thing.
Most published scientific findings are false. Take a moment to digest that.
Most published scientific findings are false. Take a moment to digest that. If you think that sounds like something a conspiracy theorist might say, you’d be right. It is one of the most oft-abused quotes in existence; and reputable outlets’ reverence for science is why you’re unlikely to have ever heard it before. However, it isn’t the claim of a conspiracy theorist. Why Most Published Research Findings Are False is the title of a paper by John Ioannidis, one of the most respected statisticians in the world, published in a leading medical journal. In fact, it’s one of the most cited papers the journal has ever published.
The reason that most published research findings are false is mainly that science is hard. Just like anyone doing very hard work, scientists make mistakes. They spend their lives working at the forefront of knowledge and it is simply not realistic to expect every finding to be correct.
Science may be flawed, but it isn’t broken. Over time, the truth comes out in the wash as findings come to be replicated, challenged, and compared as a whole in the form of a meta-analysis. Exaggeration of findings that are likely either wrong or overestimated cultivates an unwarranted distrust of science. Every time we read another story telling us researchers have come to the opposite conclusion of what was believed the previous week, people may come to the conclusion that science itself is unreliable. But science is self-correcting. It’s just that corrections rarely get the coverage that findings do.
A black-and-white view of the scientific world is dangerous. This attitude is precisely what allows things to happen like a tiny, flawed and fraudulent study destroying vaccination rates. By crying wolf every time a scientific finding is published, real scientific issues such as antibiotic resistance and global warming are ignored — and time is wasted debating and legislating dead ends.
Finally, when science is misused to serve an agenda the results are often counterproductive. Take for instance the complete failure of the former, multi-billion dollar D.A.R.E program to combat drug use, likely due to young people correctly identifying the lies they had been told and consequently doubting the truths also. The program allegedly underwent a serious overhaul in 2009 after more than a decade of scientific studies about its ineffectiveness, yet just last year claimed cannabis “is one of the most dangerous drugs on Earth” after being duped by a satirical article.
Consider for a moment if this might have happened to you. If you’re rationalizing behaviors such as smoking, binge drinking, or over-eating because you think the scientific evidence is wobbly, be under no illusion, it isn’t. Equally, if you have been scared by reports that the science on vaccines and global warming is unsound, rest assured, it isn’t. Ninety-seven percent of scientists agree about climate change and all reputable doctors and medical organizations are strongly in favor of vaccinations. Sadly, it doesn’t tend to be these scientists and doctors that get the press coverage.
Ultimately, the best way to live a life by science is to ignore most of what you read in science news and instead take a little time to grasp high quality resources. Places like the CDC website or NHS choices are a good place to find credible information. Organizations such as Sense about Science do a superb job of sorting the wheat from the chaff when it comes to science news. PrimeMind, I hope, will come to be another resource you can trust to evaluate the evidence in perspective rather than myopically reporting on individual studies, and for my part I’ll be doing my best to make sure it does.
In the end, it is up to you to be skeptical when hearing new claims. A simple rule of thumb is to never change your behavior because of a single study in the news. Instead, wait for replications to confirm results and look for reviews of the evidence based on meta-analyses. The website of the Cochrane Collaboration, who maintain a gold standard in their systematic reviews, is the best place to start your search. Most importantly, don’t be fooled by shouty news coverage of “cutting edge” or “newly published” findings. Quite simply, they are more likely to be false than true.
Simon Oxenham is a science journalist based in the U.K. He is known for exposing and debunking misinformation and keeping a watchful eye on controversial findings within the fields of psychology and neuroscience. Simon has written and blogged for The Psychologist, Nature, Scientific American, The Guardian, and Big Think, among others, and has a weekly column at New Scientist. Follow him on Twitter, Facebook, or join his mailing list.
