Peer review process improves accuracy

I think almost all the TAs I know get the same expression when a student asks why they can’t use a random website as a source for a paper. It’s somewhere between a horrified cringe and frustration.

I am not convinced that students know why asking if they can use a general website causes such a strong reaction, even when I explain the peer review process and talk about having a paper anonymously reviewed by a panel of experts. I’ve even taken to referring to it in class as training a homemade bullshit detector, but from the sometimes very confused expressions and students who persist in citing WebMD or Google answers, I get the feeling that their confusion persists.

The short answer to why you can’t use just anything as a source is accuracy: In order to be considered a useful source for research at the formal level, the source has to be as accurate as possible.

There are a few reasons why, in general, peer reviewed sources are more accurate than random sites off the Internet. The first of these has to do with the review process. When someone wants to publish a paper on something they’ve researched, they have to go through a series of steps to ensure that it can be published.

First, if they are a graduate student or not a full professor, they very likely have a committee to whom they are responsible for the data and the paper. That committee has to approve of the paper, topic, data collection methods and data in order for the paper to advance to the stage where it is able to be submitted.

Full professors are responsible to their funding committees; research is expensive, and not something most can afford out of pocket.

Once the person who wants to publish that paper has the approval to publish, and remember, this is after approval from the Institutional Research Board (IRB) at UNM and at whatever institution is funding the research, they have to find a publication which has the correct focus and in which they feel they are prestigious enough to publish. The more prestigious the magazine, the fewer papers the magazine will publish. Many publications are extraordinarily specialized and require the papers submitted to be equally specialized. Many times, the person who wants to publish also has to pay for the ability to be considered for publication, as well, so people who want to publish do not manage to publish often.

At the point where the research paper can be submitted, the publication will take a copy, remove the submitter’s name and send copies to at least three reviewers who are specialists with years of experience and professional degrees. These three reviewers are paid a stipend to render that paper down as far as they can; the default is rejection, though some places are nice enough to invite you to resend, if you make all of the lists of corrections they stipulate. For prestigious publications, all three reviewers have to agree. Some publications will allow two out of three. At any time in this process, that paper will have been looked over by at least six people, all of whom are strongly motivated by both pay and by the desire to keep their field as accurate as possible, to find inaccuracies, bad writing, bad ideas or anything they think is unworthy of the field.

At this point, at least one student will point out that he or she has heard of a time when science (the nebulous science) has changed its mind, or a scientist has been wrong. It’s true; papers can get through this process and still manage to be wrong, because everyone in the field is wrong or because the reviewers are wrong in the same way the paper is wrong.

However, the process favors accuracy. It favors accuracy both because of the multiple layers of editors and gatekeepers who are motivated to get rid of inaccuracies, and because it is modeled after the scientific method.

I always do a lecture about the scientific method when I lecture about source accuracy. I can boil it down to the idea, with the scientific method, that everything bears retesting. Scientific principles, to be considered accurate, have to be tested and retested. As long as the conditions are the same, the scientific principle has to keep working.

If it doesn’t work, it must be taken apart to find out why, and something must be learned from the process. Nothing is safe from being retested, and though scientists may be the only people capable of retesting it, they will. Often, the facilities needed to retest an idea are expensive enough to prevent the general public from being able to retest it themselves. That is, in fact, how many things in science are taught. Students are made to reproduce the effect for themselves.

Contrast this, and I say this as someone who is fond of Wikipedia, with the instant publication of one’s thoughts on Facebook, a blog or the comments on a news story. That is why your teacher keeps asking you to use peer-reviewed sources.

I wince because I care.