In its most basic sense, peer-review is when a scientist’s
research paper is evaluated by his “peers” to determine if it meets the basic
standards required for publication in a scientific journal. But this simple
definition doesn’t really explain the process, so let’s explore that in greater
depth.
To get us started, let’s define what we mean by “peer.”
We’re not talking about the kind of “peer” we think of when we say “a jury of
our peers.” In that situation, it simply means other citizens. For a jury you
often want to get some cross-section of the community – college educated and
not, male and female, white collar and blue collar employment. Everyone in the
community is your “peer” and the final jury empaneled is largely a factor of
the random order of the selection from the jury pool (plus a little selective
tweaking by lawyers for the defendant and plaintiff).
In science a “peer” is somewhat different. To be a peer you
need to have knowledge of the highly specialized subject of the paper being
reviewed. If the paper is about climate science, you obviously need to have
sufficient knowledge of climate science to be able to review the paper
effectively. Sending a climate paper to a brain surgeon for review makes no
more sense than going to a chiropractor to have your cows milked. With that in
mind, a “peer” would be another climate scientist. [Needless to say, if the
paper is about brain surgery, you would not send it to a climate scientist for
review.]
Every legitimate (i.e., peer-reviewed) journal has a staff of
editors to manage the process of review and publication. These editors will
receive research papers from the authors, determine what scientists out there
have the necessary expertise to effectively review the paper, and coordinate
the reviews and feedback to the authors. Most journals will send the paper to
three peer-reviewers, though for particularly important and/or potentially
contentious papers they may be sent to four or even five peer-reviewers. While
the editor is the go-between, the authors generally do not know who the
proposed paper has been sent to for peer review. In many cases, but not all,
the peer reviewers also don’t know the name of the author who submitted the
paper. These peers review the paper and provide their comments and recommendations:
publish as is, publish if minor errors and/or questions are addressed, publish
if major errors are addressed, or reject it because it fails to meet even the
most basic standards of veracity.
Okay, so what are these peer reviewers looking for? Mostly
they are looking to ensure that the research has been conducted, reported, and
evaluated adequately. And it has to be research. Blogs don’t normally get any
peer-review, which is why most of what you read on blogs is opinion and not
science. [But, some blogs can discuss the published science – see article here for how to discern a reliable blog from an unreliable blog.]
Peers who are reviewing a potential paper for publication as
themselves a series of questions. The first question is always, “does this
paper fit into the scope of the journal?” Since journals tend to focus in on
narrow topics, papers that don’t fit that topic shouldn’t even be considered.
Luckily, there are a myriad of journals with overlapping scopes, so a good
research paper should easily be able to find a place to be published. With that
as a given, the questions peer reviewers ask include: Is the scope of the
research study clearly presented? Do they review the prior literature on that
topic? Are the stipulated premises valid? Do they adequately explain the
methodology so others can see how they conducted the study? Do the authors
present the results in full and clearly? Do the data tables and graphics look
correct? Are the statistical procedures clearly explained and valid? Are the
conclusions reached logically derived from the data presented?
While that sounds like a lot, the idea of peer-review is not
to approve or disapprove of the research or conclusions. The goal is merely to
ensure that the paper documents and demonstrates a well-thought-out and
conducted scientific study. If it does, then it usually is published in the
journal.
Done, right?
Actually, getting through this initial peer-review should be
considered only the first step in the scientific review process. What most
people think of as peer-review just makes sure the paper appears sufficiently
documented, is a significant contribution to the science, and should be made
available to the scientific community at large through publication. But only
then – once the paper is out in what scientists call “the literature” - does it
begin to be closely scrutinized by the broader scientific community. Scientists
in the field will read it and evaluate it and, often, debate it. Are the
author’s points defensible? Does it agree or conflict with existing literature.
Does the new paper enhance our knowledge? Are there any mistakes the initial
peer-reviewers missed? Does it stand up to scrutiny?
This could go on for some time. If the paper makes important
points, especially if it changes our view of the science, it will get cited by
other papers who do follow up research. Papers cited a lot tend to be important
papers.
Many people have the impression that getting a paper
peer-reviewed means it is “science.” That isn’t exactly true. “Science” isn’t a
single scientific paper; science is the compendium of scientific papers
published on a particular topic.
This point is critical.
Scientific research usually works by increments. Individual
studies don’t investigate, for example, “is global warming happening?” That is
too big a chunk to evaluate. Instead, a study may test whether CO2 can make an
atmosphere warmer. This was done in many studies in the laboratory by many
different independent researchers. Each study is written up and published in
scientific journals. There are dozens
(actually, hundreds) of studies in the last 150 years that
examine this exact same question using many different methodologies, all of
which are published in journals. The sum total of all of those studies – each
looking at the same thing from different angles – tell us without any doubt
that yes, CO2 can make an atmosphere warmer.
Other studies may look at “how much warmer?” Or “if it works
in the lab, does it also work in the global atmosphere?” Or any number of
related questions. Still other studies look at the effect of clouds, the
impacts of a warmer climate on extreme weather events, the acidification of the
oceans, etc. All researched, all published, all scrutinized by dozens or
hundreds or even thousands of other scientists. Eventually the data are so
overwhelming and so clear and undeniable that everyone recognizes the fact of
the science. That is the case for evolution, gravity, and yes, man-made climate
change.
One more aspect of peer-review is important – it never
stops. Scientists continue to conduct new studies to examine new questions. The
new results are assessed in the context of all the other results – do they
agree or disagree with our current understanding? Do they enhance our
knowledge? Do they change our understanding of the science? All of these
questions are revisited with every new study and every new chance at
peer-review. In the case of climate change, new studies overwhelmingly confirm
that human activity is warming the climate system.
Finally, I mentioned earlier that the initial peer-review
process (deciding whether to publish or not) doesn’t make a final determination
about the defensibility of the paper. That comes afterward, when any questions
from other scientists will have to be addressed by the authors. That is part of
the process. But sometimes, a paper gets through peer-review that isn’t
supportable even on its face. In the next post I’ll talk about what happens
when unsupportable papers get published. I’ll also talk about why some journals
have intentionally low standards or “pal-review” systems. Lastly, I’ll talk
about the challenges created by a new breed of journals – the “pay-per-publish”
type that raises questions about the integrity of the “non-peer-reviewed”
publishing process.
[Note: The above is Part 1 of a series on peer-review, how it works and doesn't work, and how some people try to influence the public through bypassing peer-review. Click on the links to read Part 2, Part 3, and Part 4.]
[Note: The above is Part 1 of a series on peer-review, how it works and doesn't work, and how some people try to influence the public through bypassing peer-review. Click on the links to read Part 2, Part 3, and Part 4.]
[Note: Peer-review graphic can be seen larger at http://undsci.berkeley.edu/article/howscienceworks_16]
