• Login
  • Register
The Daily Sceptic
No Result
View All Result
  • Articles
  • About
  • Archive
    • ARCHIVE
    • NEWS ROUND-UPS
  • Podcasts
  • Newsletter
  • Premium
  • Donate
  • Log In
The Daily Sceptic
No Result
View All Result

Status Bias in Peer Review

by Noah Carl
20 September 2022 7:00 AM

Almost all scientific papers go through something called ‘peer review’ before they’re published.

This usually works as follows: the paper is sent to a scientific journal; the editor of that journal selects two or three reviewers whom he or she considers experts in the relevant subject matter; each reviewer reads the paper and submits a report indicating whether it should be accepted, rejected or revised; the editor then makes a decision based the reviewers’ reports.

Most journals use ‘double blind’ reviewing, where the reviewers don’t know the identity of the authors, and the authors don’t know the identity of the reviewers. Some journals use ‘single blind’ reviewing, where the reviewers know the identity of the authors, or vice versa. And a small number of journals use ‘open peer review’, where everyone’s identity is known to everyone else.

Even when journals use double blind reviewing, it is often still possible for reviewers to work out who the authors are. The latter may have already may have presented the paper at a conference or published a ‘preprint’ online. Sometimes it is possible to work out who the authors purely from the content of the paper (e.g., because it builds on previous research they have done).

In an ideal world, reviewers’ recommendations would be based solely on objective criteria, like the suitability of the research design and the absence of errors in analysis. Yet evidence (and the personal experience of many academics) suggests this is rarely the case. Various forms of bias afflict the reviewing process.

In a new paper (which hasn’t yet been peer-reviewed!) Juergen Huber and colleagues explore one particular form of bias: ‘status bias’. Are reviewers more likely to recommend ‘revise’ or ‘accept when the authors are big names in the field than when they’re complete nobodies?

Huber and colleagues carried out a field experiment, working with the editor of a major economics journal. A paper was submitted to the journal which had two authors: one a Nobel Prize winner in economics, the other an unknown research associate. Like the journal’s editor, they were both ‘in on’ the study.

The paper was then sent out to 3,300 potential reviewers, in order to get a large enough sample size. There were three different treatments.

In treatment ‘AL’, the unknown research associate was given as the paper’s corresponding author (so this was single blind). In ‘AA’, no corresponding author was given (so this was double blind). And in ‘AH’, the Nobel Prize winner was given as the corresponding author (this was single blind again).

Huber and colleagues’ main result is shown in the chart below. Each coloured bar represents the distribution of reviewers’ recommendations in a particular treatment.

The distribution of reviewers’ recommendations in three different treatments.

As you can see, reviewers were far more likely to say the paper should be rejected when the unknown research associate was given as the corresponding author. In ‘AL’, 65% of reviewers recommended ‘reject’, compared to only 23% in ‘AH’. So you’re about one third as likely to have your paper rejected if you’re a Nobel Prize winner.

Needless to say, this isn’t how science is meant to work. Rather than judging papers on their own merits, most reviewers are content to employ heuristics like “work by a Nobel Prize winner must be good” or, indeed, “work by an unknown research associate must be bad”.

The pandemic showed what can happen when defer too much to credentialed ‘experts’. And it’s not just politicians and laypeople that fall into this traps – so do scientists themselves.

Tags: ExpertsPeer reviewThe Science

Donate

We depend on your donations to keep this site going. Please give what you can.

Donate Today

Comment on this Article

You’ll need to set up an account to comment if you don’t already have one. We ask for a minimum donation of £5 if you'd like to make a comment or post in our Forums.

Sign Up
Previous Post

News Round-Up

Next Post

The One-Sided Media Coverage of the Queen’s Death Was Painfully Reminiscent of Covid Propaganda For Many

Subscribe
Login
Notify of
Please log in to comment

To join in with the discussion please make a donation to The Daily Sceptic.

Profanity and abuse will be removed and may lead to a permanent ban.

3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Marcus Aurelius knew
Marcus Aurelius knew
2 years ago

“In February 1616, the Inquisition assembled a committee of theologians, known as qualifiers, who delivered their unanimous report condemning heliocentrism as “foolish and absurd in philosophy, and formally heretical since it explicitly contradicts in many places the sense of Holy Scripture.” Wikipedia

I note that it was only relatively recently (1822) that the Catholic Church removed works proposing heliocentrism from its Index of Forbidden Books. But has it ever actually accepted heliocentrism?

The problem is as old as the hills…

Visited Toruń, Poland two weeks ago. Copernicus was a very brave, intelligent, humble chap.

Last edited 2 years ago by Marcus Aurelius knew
19
0
True Spirit of America Party
True Spirit of America Party
2 years ago
Reply to  Marcus Aurelius knew

It was not until 1992 I think that the Church finally officially admitted that Galileo was right all along.

8
0
True Spirit of America Party
True Spirit of America Party
2 years ago

Pal review is more like it.

16
0

NEWSLETTER

View today’s newsletter

To receive our latest news in the form of a daily email, enter your details here:

DONATE

PODCAST

Episode 36 of the Sceptic: Karl Williams on Starmer’s Phoney Immigration Crackdown, Dan Hitchens on the Assisted Suicide Bill and Tom Jones on Reform’s Local Council Challenge

by Richard Eldred
16 May 2025
0

LISTED ARTICLES

  • Most Read
  • Most Commented
  • Editor’s Picks

Chinese ‘Kill Switches’ Found in US Solar Farms

15 May 2025
by Will Jones

News Round-Up

16 May 2025
by Richard Eldred

Spy Agency Report on the Alleged “Extremism” of AfD Turns Out to Be So Stupid That it Destroys all Momentum for Banning the Party

16 May 2025
by Eugyppius

The Folly of Solar – a Dot on the Horizon Versus a Blight on the Land

16 May 2025
by Ben Pile

Civil Servants Threaten to Strike Over Trans Ban in Women’s Lavatories

16 May 2025
by Will Jones

The Folly of Solar – a Dot on the Horizon Versus a Blight on the Land

29

Civil Servants Threaten to Strike Over Trans Ban in Women’s Lavatories

26

Spy Agency Report on the Alleged “Extremism” of AfD Turns Out to Be So Stupid That it Destroys all Momentum for Banning the Party

19

News Round-Up

18

Chinese ‘Kill Switches’ Found in US Solar Farms

27

Trump’s Lesson in Remedial Education

16 May 2025
by Dr James Allan

Spy Agency Report on the Alleged “Extremism” of AfD Turns Out to Be So Stupid That it Destroys all Momentum for Banning the Party

16 May 2025
by Eugyppius

The Folly of Solar – a Dot on the Horizon Versus a Blight on the Land

16 May 2025
by Ben Pile

Renaud Camus on the Destruction of Western Education

15 May 2025
by Dr Nicholas Tate

‘Why Can’t We Talk About This?’

15 May 2025
by Richard Eldred

POSTS BY DATE

September 2022
M T W T F S S
 1234
567891011
12131415161718
19202122232425
2627282930  
« Aug   Oct »

SOCIAL LINKS

Free Speech Union
  • Home
  • About us
  • Donate
  • Privacy Policy

Facebook

  • X

Instagram

RSS

Subscribe to our newsletter

© Skeptics Ltd.

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms below to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Articles
  • About
  • Archive
    • ARCHIVE
    • NEWS ROUND-UPS
  • Podcasts
  • Newsletter
  • Premium
  • Donate
  • Log In

© Skeptics Ltd.

wpDiscuz
You are going to send email to

Move Comment
Perfecty
Do you wish to receive notifications of new articles?
Notifications preferences