Thursday, February 19, 2015

How peer-review works…and doesn’t work (Part 3: Abusing the system)

This is a continuation of the series on how peer-review works…and doesn’t work. Part 1 looked at the basics of how the peer-review process works for scientific papers - what it does, and what it doesn't do. You can read the entire Part 1 article here. Part 2 looked at what happens when peer-review goes wrong. You can read the entire Part 2 article here. Now we’ll take a look at some cases where the peer-review system has been abused. 

Before starting Part 3, however, it must be stressed that any inadequacies so far discussed are exceptions to the rule. Peer-review almost always does what it is supposed to do – a first screen to make sure papers represent legitimate research and are fully documented so that they can be assessed by the larger scientific community. It’s rare that peer-review “fails” (see Part 2). It’s even rarer that papers are retracted once they are published. A study published in 2012 examined the 2047 retractions of papers indexed in the PubMed database (mostly biotechnology papers). That sounds like a lot until you realize that this was out of over 21 million published papers in that database, meaning less than 0.01% of published papers were retracted. Retraction is rare even though the bar for retracting papers has been lowered (i.e., it's much easier and faster to retract now than previously).
That said, let’s look at the cases where papers have been published that probably shouldn’t have been. The new problem of “open access journals,” i.e., those journals who publish for a fee, was mentioned in Part 2. The biggest concern here is that some of these “journals” are simply predatory publishers that will post online anything that is sent to them as long as the fee is paid. These predatory journals will likely disappear as people refuse to be associate with them, especially since they obviously aren’t really peer-reviewed. So while they may be a big headache right now, likely they will weed out the bad eggs through, not ironically, peer pressure. 

Which gets us to the real problem. The following examples highlight some of what can happen when unscrupulous people try to take advantage of the system. 

The most famous example of “pal review” as discussed in Part 2 is the publication of a climate related paper by Soon and Baliunas in the journal Climate Research in 2003. The paper was shuttled through the review process by fellow climate denier Chris de Freitas, an editor for the journal. Once published, the paper was roundly criticized by the scientific community as unsupportable on its face. Further review revealed that Soon and Baliunas were funded by the fossil fuel industry, that the conclusions stated were inconsistent with their own data (which were inconsistent with reality), and that de Frietas had a history of pushing through papers by climate deniers despite their obvious failings. Details of the controversy can be read here. Since then, Soon and a small group of lobbyist-associated authors have been implicated in a series of questionable papers that misrepresent the science. Often these papers are published in a journal called Energy and Environment, a non-science pal-review type of journal where the editor has acknowledged papers are published based on political motives. 

Following publication of the Soon and Baliunas paper described above, and also in one or two other cases where apparently fraudulent papers were published in peer-reviewed journals, senior editors chose to resign. While reputations of any scientists involved can be severely damaged, for some this doesn’t appear to matter much as long as the lobbyist funding continues (e.g., Soon was recently accused of violating basic ethics conventions by failing to disclose his fossil fuel industry funding in a paper he co-authored with the usual band of climate deniers). 

There isn’t much that can be done about such papers other than to keep strengthening peer-review standards, a difficult proposition given the thousands of journals that now compete for papers to publish. Sometimes the papers are retracted, but as noted above, retractions are rare, though increasing.  

This latter point can actually work against legitimate scientists. In the past, scientific papers were scrutinized and critiqued by other scientists, and that feedback helped move the science along. Now the papers are more accessible to the general public through blogs, the public is more likely to get a "spun" version of the paper than the actual science. While press releases by the scientific organizations may be poorly worded, the real problem is when bloggers, either intentionally or unintentionally, get the gist of the paper's findings wrong. So the public may be misinformed. Worse, the papers are read by political and lobbying interests, which would be okay if they honestly evaluated the science. But that isn’t the case. Most political operatives and lobbyists have a particular policy view and are not hesitant to misrepresent the science if they feel doing so will help them achieve their preferred policy action – which in most cases is no action at all. These operatives and lobbyists can exert tremendous pressure on journals that, at least in one recent case, can lead to legitimate, scientifically robust, papers being retracted solely because the journal feared an expensive legal battle with lobbyists. This sets a dangerous precedent. 

In addition, there are many cases of politicians saying things about science that are not scientific. Senator James Inhofe is notorious for arguing that the science of man-made climate change is all a hoax, originally basing this politically convenient opinion on the 2003 Soon and Baliunas paper, which many suggest was the main motivation for the paper being funded by the petroleum industry. Not surprisingly, Inhofe’s home state of Oklahoma is highly dependent on the oil and gas industry and that industry routinely lavishes upon him significant campaign funding. This is true of other politicians as well. And yes, health and environmental advocacy groups also financially support their preferred politicians and feed them information that supports their advocacy. The main difference is that health and environmental lobbyists generally pressure politicians to listen to the scientists while fossil fuel lobbyists generally pressure politicians to listen to, well, the fossil fuel lobbyists and their small cadre of associated scientists who disagree with the vast overriding consensus of the science. 

But that’s a topic for another post.  

To recap, the last three weeks have taken a look at the peer-review process – what it is, and what it isn’t. We’ve looked at some ways that peer-review can “fail,” and some ways that people have abused the process. Due to the space limitations of a blog format, these discussions are necessarily incomplete. The links provide more detail on some of the points being made, but there are many others that could also be discussed in greater depth. The main points to understand are that peer-review is merely the first step in the scientific evaluation process, and only after publication can the greater scientific community scrutinize the studies being presented. Sometimes bad papers get published, but most of the time they are inconsequential. Attempts at fraud do happen, and while relatively rare, can have significant impacts (e.g., see Andrew Wakefield).
Overall, peer-review works, and is necessary. There are challenges for the future because of predatory practices related to the “open access” nature of the worldwide web, but these are likely to be worked out so that some combination of public access and quality assurance can be achieved. 

In Part 4 we'll take a look at how some papers that might have been inconsequential in the past can now be artificially elevated into a level of importance they don't merit. We'll explore the role of the internet in making this happen, both for good and for evil.

[Note: Peer-review graphic can be seen larger at


Victor Venema said...

There isn’t much that can be done about such papers other than to keep strengthening peer-review standards, a difficult proposition given the thousands of journals that now compete for papers to publish. Sometimes the papers are retracted, but as noted above, retractions are rare, though increasing. This latter point can actually work against legitimate scientists.

Being a "legitimate" scientist, I would argue that we should not make peer review more strict. It is just a filter, some bad stuff gets through, but on average the quality is better afterwards. If we make the review more strict, we risk that good and especially very unconventional ideas will find it hard to get published. And like you write yourself, it is impossible to make peer review so strict that only "right" papers get published, whatever that means. The papers should just be worthwhile to read.

For science it would thus be best to keep the standards. What would not hurt would be to apply the standards more consistently. That is hard to do because the reviews are written by volunteers, but that would be worthwhile.

That has as consequence that we have to tell society and journalists that they should never rely on a single new paper. It only becomes science after being digested by the scientific community, that takes time. Great to see how this blog series contributes to that.

The Dake Page said...

You make excellent points and I didn't mean to suggest that legitimate papers that have unconventional ideas be restricted. My suggestion that peer-review standards be strengthened is more focused on keeping the process free from abuse such as that described, and ensuring that some sort of peer-review is included in the new open-access journals.

This last point is probably the greater issue. Simply getting gibberish published because you paid the fee gets the notice, but there could be many "propaganda" papers also being published this way. Part 4 takes a look at papers that would usually be ignored because they lack robustness now, in the age of universal access of "information" (whether real or not) on the internet, inconsequential papers can take on undue importance. Especially when lobbyists (pro- or anti-industry) promote them.

Your last paragraph captures exactly a point that scientists need to keep emphasizing to journalists - one paper does not define a body of science, and most papers focus on one small aspect, a piece of a very large puzzle that only makes sense when you look at the puzzle as a whole. Journalists (and to an even greater extent, bloggers) routinely take some tiny paper and elevate to seminal without merit.

More in Part 4.

S. Lawrence Parrish said...

Thanks for this.