• Login
  • Register
The Daily Sceptic
No Result
View All Result
  • Articles
  • About
  • Archive
    • ARCHIVE
    • NEWS ROUND-UPS
  • Podcasts
  • Newsletter
  • Premium
  • Donate
  • Log In
The Daily Sceptic
No Result
View All Result

Status Bias in Peer Review

by Noah Carl
20 September 2022 7:00 AM

Almost all scientific papers go through something called ‘peer review’ before they’re published.

This usually works as follows: the paper is sent to a scientific journal; the editor of that journal selects two or three reviewers whom he or she considers experts in the relevant subject matter; each reviewer reads the paper and submits a report indicating whether it should be accepted, rejected or revised; the editor then makes a decision based the reviewers’ reports.

Most journals use ‘double blind’ reviewing, where the reviewers don’t know the identity of the authors, and the authors don’t know the identity of the reviewers. Some journals use ‘single blind’ reviewing, where the reviewers know the identity of the authors, or vice versa. And a small number of journals use ‘open peer review’, where everyone’s identity is known to everyone else.

Even when journals use double blind reviewing, it is often still possible for reviewers to work out who the authors are. The latter may have already may have presented the paper at a conference or published a ‘preprint’ online. Sometimes it is possible to work out who the authors purely from the content of the paper (e.g., because it builds on previous research they have done).

In an ideal world, reviewers’ recommendations would be based solely on objective criteria, like the suitability of the research design and the absence of errors in analysis. Yet evidence (and the personal experience of many academics) suggests this is rarely the case. Various forms of bias afflict the reviewing process.

In a new paper (which hasn’t yet been peer-reviewed!) Juergen Huber and colleagues explore one particular form of bias: ‘status bias’. Are reviewers more likely to recommend ‘revise’ or ‘accept when the authors are big names in the field than when they’re complete nobodies?

Huber and colleagues carried out a field experiment, working with the editor of a major economics journal. A paper was submitted to the journal which had two authors: one a Nobel Prize winner in economics, the other an unknown research associate. Like the journal’s editor, they were both ‘in on’ the study.

The paper was then sent out to 3,300 potential reviewers, in order to get a large enough sample size. There were three different treatments.

In treatment ‘AL’, the unknown research associate was given as the paper’s corresponding author (so this was single blind). In ‘AA’, no corresponding author was given (so this was double blind). And in ‘AH’, the Nobel Prize winner was given as the corresponding author (this was single blind again).

Huber and colleagues’ main result is shown in the chart below. Each coloured bar represents the distribution of reviewers’ recommendations in a particular treatment.

The distribution of reviewers’ recommendations in three different treatments.

As you can see, reviewers were far more likely to say the paper should be rejected when the unknown research associate was given as the corresponding author. In ‘AL’, 65% of reviewers recommended ‘reject’, compared to only 23% in ‘AH’. So you’re about one third as likely to have your paper rejected if you’re a Nobel Prize winner.

Needless to say, this isn’t how science is meant to work. Rather than judging papers on their own merits, most reviewers are content to employ heuristics like “work by a Nobel Prize winner must be good” or, indeed, “work by an unknown research associate must be bad”.

The pandemic showed what can happen when defer too much to credentialed ‘experts’. And it’s not just politicians and laypeople that fall into this traps – so do scientists themselves.

Tags: ExpertsPeer reviewThe Science

Donate

We depend on your donations to keep this site going. Please give what you can.

Donate Today

Comment on this Article

You’ll need to set up an account to comment if you don’t already have one. We ask for a minimum donation of £5 if you'd like to make a comment or post in our Forums.

Sign Up
Previous Post

News Round-Up

Next Post

The One-Sided Media Coverage of the Queen’s Death Was Painfully Reminiscent of Covid Propaganda For Many

Subscribe
Login
Notify of
Please log in to comment

To join in with the discussion please make a donation to The Daily Sceptic.

Profanity and abuse will be removed and may lead to a permanent ban.

3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments

NEWSLETTER

View today’s newsletter

To receive our latest news in the form of a daily email, enter your details here:

DONATE

PODCAST

The Sceptic | Episode 40: Rob Bates on Stopping Britain Becoming Majority-Minority, Tilak Doshi on Trump vs Green Blob and Mario Trabucco on Osborne’s Elgin Marbles Betrayal

by Richard Eldred
13 June 2025
1

LISTED ARTICLES

  • Most Read
  • Most Commented
  • Editor’s Picks

Why Is Doctor Who So Gay? Because so Are His Current Creators

16 June 2025
by Steven Tucker

News Round-Up

16 June 2025
by Richard Eldred

How Covid Killed the Rule of Law

16 June 2025
by Nick McBride

Big Brother Lives Next Door

16 June 2025
by Matthew Corrigan

Pakistan “Will Nuke Israel if Netanyahu Uses Nuclear Weapons Against Tehran”, Iran Claims

16 June 2025
by Will Jones

Establishment in Denial Over Ethnicity of Grooming Gangs, Report Finds

30

Why Is Doctor Who So Gay? Because so Are His Current Creators

23

How Covid Killed the Rule of Law

16

How Nuclear Power Might Save The Day

15

News Round-Up

13

Big Brother Lives Next Door

16 June 2025
by Matthew Corrigan

Please Give Generously to Keep the Daily Sceptic Afloat

16 June 2025
by Toby Young

How Covid Killed the Rule of Law

16 June 2025
by Nick McBride

Why Is Doctor Who So Gay? Because so Are His Current Creators

16 June 2025
by Steven Tucker

The War in Ukraine Has Shattered the West’s Digital-Age Delusions

15 June 2025
by David Betz and Michael Rainsborough

POSTS BY DATE

September 2022
M T W T F S S
 1234
567891011
12131415161718
19202122232425
2627282930  
« Aug   Oct »

SOCIAL LINKS

Free Speech Union

NEWSLETTER

View today’s newsletter

To receive our latest news in the form of a daily email, enter your details here:

POSTS BY DATE

September 2022
M T W T F S S
 1234
567891011
12131415161718
19202122232425
2627282930  
« Aug   Oct »

DONATE

LISTED ARTICLES

  • Most Read
  • Most Commented
  • Editor’s Picks

Why Is Doctor Who So Gay? Because so Are His Current Creators

16 June 2025
by Steven Tucker

News Round-Up

16 June 2025
by Richard Eldred

How Covid Killed the Rule of Law

16 June 2025
by Nick McBride

Big Brother Lives Next Door

16 June 2025
by Matthew Corrigan

Pakistan “Will Nuke Israel if Netanyahu Uses Nuclear Weapons Against Tehran”, Iran Claims

16 June 2025
by Will Jones

Establishment in Denial Over Ethnicity of Grooming Gangs, Report Finds

30

Why Is Doctor Who So Gay? Because so Are His Current Creators

23

How Covid Killed the Rule of Law

16

How Nuclear Power Might Save The Day

15

News Round-Up

13

Big Brother Lives Next Door

16 June 2025
by Matthew Corrigan

Please Give Generously to Keep the Daily Sceptic Afloat

16 June 2025
by Toby Young

How Covid Killed the Rule of Law

16 June 2025
by Nick McBride

Why Is Doctor Who So Gay? Because so Are His Current Creators

16 June 2025
by Steven Tucker

The War in Ukraine Has Shattered the West’s Digital-Age Delusions

15 June 2025
by David Betz and Michael Rainsborough

SOCIAL LINKS

Free Speech Union
  • Home
  • About us
  • Donate
  • Privacy Policy

Facebook

  • X

Instagram

RSS

Subscribe to our newsletter

© Skeptics Ltd.

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms below to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Articles
  • About
  • Archive
    • ARCHIVE
    • NEWS ROUND-UPS
  • Podcasts
  • Newsletter
  • Premium
  • Donate
  • Log In

© Skeptics Ltd.

wpDiscuz
You are going to send email to

Move Comment