How peer-review works…and doesn’t work (Part 1)

You’ll see the term “peer-review” a lot on these pages, as well as on both scientific and denialist blogs, and in the media. Unfortunately, the term is often used incorrectly, sometimes on purpose, but mostly because the process isn’t clear to the public. This extended post will take a shot at explaining what peer-review is…and what it isn’t. We’ll talk about how it works…and why it sometimes doesn’t work.

In its most basic sense, peer-review is when a scientist’s research paper is evaluated by his “peers” to determine if it meets the basic standards required for publication in a scientific journal. But this simple definition doesn’t really explain the process, so let’s explore that in greater depth.

To get us started, let’s define what we mean by “peer.” We’re not talking about the kind of “peer” we think of when we say “a jury of our peers.” In that situation, it simply means other citizens. For a jury you often want to get some cross-section of the community – college educated and not, male and female, white collar and blue collar employment. Everyone in the community is your “peer” and the final jury empaneled is largely a factor of the random order of the selection from the jury pool (plus a little selective tweaking by lawyers for the defendant and plaintiff).

In science a “peer” is somewhat different. To be a peer you need to have knowledge of the highly specialized subject of the paper being reviewed. If the paper is about climate science, you obviously need to have sufficient knowledge of climate science to be able to review the paper effectively. Sending a climate paper to a brain surgeon for review makes no more sense than going to a chiropractor to have your cows milked. With that in mind, a “peer” would be another climate scientist. [Needless to say, if the paper is about brain surgery, you would not send it to a climate scientist for review.]

Every legitimate (i.e., peer-reviewed) journal has a staff of editors to manage the process of review and publication. These editors will receive research papers from the authors, determine what scientists out there have the necessary expertise to effectively review the paper, and coordinate the reviews and feedback to the authors. Most journals will send the paper to three peer-reviewers, though for particularly important and/or potentially contentious papers they may be sent to four or even five peer-reviewers. While the editor is the go-between, the authors generally do not know who the proposed paper has been sent to for peer review. In many cases, but not all, the peer reviewers also don’t know the name of the author who submitted the paper. These peers review the paper and provide their comments and recommendations: publish as is, publish if minor errors and/or questions are addressed, publish if major errors are addressed, or reject it because it fails to meet even the most basic standards of veracity. 

Okay, so what are these peer reviewers looking for? Mostly they are looking to ensure that the research has been conducted, reported, and evaluated adequately. And it has to be research. Blogs don’t normally get any peer-review, which is why most of what you read on blogs is opinion and not science. [But, some blogs can discuss the published science – see article here for how to discern a reliable blog from an unreliable blog.]

Peers who are reviewing a potential paper for publication as themselves a series of questions. The first question is always, “does this paper fit into the scope of the journal?” Since journals tend to focus in on narrow topics, papers that don’t fit that topic shouldn’t even be considered. Luckily, there are a myriad of journals with overlapping scopes, so a good research paper should easily be able to find a place to be published. With that as a given, the questions peer reviewers ask include: Is the scope of the research study clearly presented? Do they review the prior literature on that topic? Are the stipulated premises valid? Do they adequately explain the methodology so others can see how they conducted the study? Do the authors present the results in full and clearly? Do the data tables and graphics look correct? Are the statistical procedures clearly explained and valid? Are the conclusions reached logically derived from the data presented?

While that sounds like a lot, the idea of peer-review is not to approve or disapprove of the research or conclusions. The goal is merely to ensure that the paper documents and demonstrates a well-thought-out and conducted scientific study. If it does, then it usually is published in the journal.

Done, right?

Actually, getting through this initial peer-review should be considered only the first step in the scientific review process. What most people think of as peer-review just makes sure the paper appears sufficiently documented, is a significant contribution to the science, and should be made available to the scientific community at large through publication. But only then – once the paper is out in what scientists call “the literature” - does it begin to be closely scrutinized by the broader scientific community. Scientists in the field will read it and evaluate it and, often, debate it. Are the author’s points defensible? Does it agree or conflict with existing literature. Does the new paper enhance our knowledge? Are there any mistakes the initial peer-reviewers missed? Does it stand up to scrutiny?

This could go on for some time. If the paper makes important points, especially if it changes our view of the science, it will get cited by other papers who do follow up research. Papers cited a lot tend to be important papers. 

Many people have the impression that getting a paper peer-reviewed means it is “science.” That isn’t exactly true. “Science” isn’t a single scientific paper; science is the compendium of scientific papers published on a particular topic.

This point is critical.

Scientific research usually works by increments. Individual studies don’t investigate, for example, “is global warming happening?” That is too big a chunk to evaluate. Instead, a study may test whether CO2 can make an atmosphere warmer. This was done in many studies in the laboratory by many different independent researchers. Each study is written up and published in scientific journals.  There are dozens (actually, hundreds) of studies in the last 150 years that examine this exact same question using many different methodologies, all of which are published in journals. The sum total of all of those studies – each looking at the same thing from different angles – tell us without any doubt that yes, CO2 can make an atmosphere warmer. 

Other studies may look at “how much warmer?” Or “if it works in the lab, does it also work in the global atmosphere?” Or any number of related questions. Still other studies look at the effect of clouds, the impacts of a warmer climate on extreme weather events, the acidification of the oceans, etc. All researched, all published, all scrutinized by dozens or hundreds or even thousands of other scientists. Eventually the data are so overwhelming and so clear and undeniable that everyone recognizes the fact of the science. That is the case for evolution, gravity, and yes, man-made climate change.

One more aspect of peer-review is important – it never stops. Scientists continue to conduct new studies to examine new questions. The new results are assessed in the context of all the other results – do they agree or disagree with our current understanding? Do they enhance our knowledge? Do they change our understanding of the science? All of these questions are revisited with every new study and every new chance at peer-review. In the case of climate change, new studies overwhelmingly confirm that human activity is warming the climate system.

Finally, I mentioned earlier that the initial peer-review process (deciding whether to publish or not) doesn’t make a final determination about the defensibility of the paper. That comes afterward, when any questions from other scientists will have to be addressed by the authors. That is part of the process. But sometimes, a paper gets through peer-review that isn’t supportable even on its face. In the next post I’ll talk about what happens when unsupportable papers get published. I’ll also talk about why some journals have intentionally low standards or “pal-review” systems. Lastly, I’ll talk about the challenges created by a new breed of journals – the “pay-per-publish” type that raises questions about the integrity of the “non-peer-reviewed” publishing process.

[Note: The above is Part 1 of a series on peer-review, how it works and doesn't work, and how some people try to influence the public through bypassing peer-review. Click on the links to read Part 2, Part 3, and Part 4.]

[Note: Peer-review graphic can be seen larger at http://undsci.berkeley.edu/article/howscienceworks_16]