How peer-review works…and doesn’t work (Part 3: Abusing the system)

This is a continuation of the series on how peer-review works…and doesn’t work. Part 1 looked at the basics of how the peer-review process works for scientific papers - what it does, and what it doesn't do. You can read the entire Part 1 article here. Part 2 looked at what happens when peer-review goes wrong. You can read the entire Part 2 article here. Now we’ll take a look at some cases where the peer-review system has been abused. 

Before starting Part 3, however, it must be stressed that any inadequacies so far discussed are exceptions to the rule. Peer-review almost always does what it is supposed to do – a first screen to make sure papers represent legitimate research and are fully documented so that they can be assessed by the larger scientific community. It’s rare that peer-review “fails” (see Part 2). It’s even rarer that papers are retracted once they are published. A study published in 2012 examined the 2047 retractions of papers indexed in the PubMed database (mostly biotechnology papers). That sounds like a lot until you realize that this was out of over 21 million published papers in that database, meaning less than 0.01% of published papers were retracted. Retraction is rare even though the bar for retracting papers has been lowered (i.e., it's much easier and faster to retract now than previously).

  

That said, let’s look at the cases where papers have been published that probably shouldn’t have been. The new problem of “open access journals,” i.e., those journals who publish for a fee, was mentioned in Part 2. The biggest concern here is that some of these “journals” are simply predatory publishers that will post online anything that is sent to them as long as the fee is paid. These predatory journals will likely disappear as people refuse to be associate with them, especially since they obviously aren’t really peer-reviewed. So while they may be a big headache right now, likely they will weed out the bad eggs through, not ironically, peer pressure. 

Which gets us to the real problem. The following examples highlight some of what can happen when unscrupulous people try to take advantage of the system. 

The most famous example of “pal review” as discussed in Part 2 is the publication of a climate related paper by Soon and Baliunas in the journal Climate Research in 2003. The paper was shuttled through the review process by fellow climate denier Chris de Freitas, an editor for the journal. Once published, the paper was roundly criticized by the scientific community as unsupportable on its face. Further review revealed that Soon and Baliunas were funded by the fossil fuel industry, that the conclusions stated were inconsistent with their own data (which were inconsistent with reality), and that de Frietas had a history of pushing through papers by climate deniers despite their obvious failings. Details of the controversy can be read here. Since then, Soon and a small group of lobbyist-associated authors have been implicated in a series of questionable papers that misrepresent the science. Often these papers are published in a journal called Energy and Environment, a non-science pal-review type of journal where the editor has acknowledged papers are published based on political motives. 

Following publication of the Soon and Baliunas paper described above, and also in one or two other cases where apparently fraudulent papers were published in peer-reviewed journals, senior editors chose to resign. While reputations of any scientists involved can be severely damaged, for some this doesn’t appear to matter much as long as the lobbyist funding continues (e.g., Soon was recently accused of violating basic ethics conventions by failing to disclose his fossil fuel industry funding in a paper he co-authored with the usual band of climate deniers). 

There isn’t much that can be done about such papers other than to keep strengthening peer-review standards, a difficult proposition given the thousands of journals that now compete for papers to publish. Sometimes the papers are retracted, but as noted above, retractions are rare, though increasing.  

This latter point can actually work against legitimate scientists. In the past, scientific papers were scrutinized and critiqued by other scientists, and that feedback helped move the science along. Now the papers are more accessible to the general public through blogs, the public is more likely to get a "spun" version of the paper than the actual science. While press releases by the scientific organizations may be poorly worded, the real problem is when bloggers, either intentionally or unintentionally, get the gist of the paper's findings wrong. So the public may be misinformed. Worse, the papers are read by political and lobbying interests, which would be okay if they honestly evaluated the science. But that isn’t the case. Most political operatives and lobbyists have a particular policy view and are not hesitant to misrepresent the science if they feel doing so will help them achieve their preferred policy action – which in most cases is no action at all. These operatives and lobbyists can exert tremendous pressure on journals that, at least in one recent case, can lead to legitimate, scientifically robust, papers being retracted solely because the journal feared an expensive legal battle with lobbyists. This sets a dangerous precedent. 

In addition, there are many cases of politicians saying things about science that are not scientific. Senator James Inhofe is notorious for arguing that the science of man-made climate change is all a hoax, originally basing this politically convenient opinion on the 2003 Soon and Baliunas paper, which many suggest was the main motivation for the paper being funded by the petroleum industry. Not surprisingly, Inhofe’s home state of Oklahoma is highly dependent on the oil and gas industry and that industry routinely lavishes upon him significant campaign funding. This is true of other politicians as well. And yes, health and environmental advocacy groups also financially support their preferred politicians and feed them information that supports their advocacy. The main difference is that health and environmental lobbyists generally pressure politicians to listen to the scientists while fossil fuel lobbyists generally pressure politicians to listen to, well, the fossil fuel lobbyists and their small cadre of associated scientists who disagree with the vast overriding consensus of the science. 

But that’s a topic for another post.  

To recap, the last three weeks have taken a look at the peer-review process – what it is, and what it isn’t. We’ve looked at some ways that peer-review can “fail,” and some ways that people have abused the process. Due to the space limitations of a blog format, these discussions are necessarily incomplete. The links provide more detail on some of the points being made, but there are many others that could also be discussed in greater depth. The main points to understand are that peer-review is merely the first step in the scientific evaluation process, and only after publication can the greater scientific community scrutinize the studies being presented. Sometimes bad papers get published, but most of the time they are inconsequential. Attempts at fraud do happen, and while relatively rare, can have significant impacts (e.g., see Andrew Wakefield).

  

Overall, peer-review works, and is necessary. There are challenges for the future because of predatory practices related to the “open access” nature of the worldwide web, but these are likely to be worked out so that some combination of public access and quality assurance can be achieved. 

In Part 4 we'll take a look at how some papers that might have been inconsequential in the past can now be artificially elevated into a level of importance they don't merit. We'll explore the role of the internet in making this happen, both for good and for evil.

[Note: Peer-review graphic can be seen larger at http://undsci.berkeley.edu/article/howscienceworks_16]