• Login
  • Register
The Daily Sceptic
No Result
View All Result
  • Articles
  • About
  • Archive
    • ARCHIVE
    • NEWS ROUND-UPS
  • Podcasts
  • Newsletter
  • Premium
  • Donate
  • Log In
The Daily Sceptic
No Result
View All Result

Can We Trust the Science of Ecology?

by Noah Carl
8 January 2024 11:00 AM

Findings reported by ecologists are hugely influential in policy debates over agriculture, pollution, climate change, conservation and biodiversity. But can we trust them? A recent paper suggests that some scepticism is in order.

Kaitlin Kimmel and colleagues set out to examine the prevalence of “questionable research practices” in the field of ecology. QRPs, as they are known, include things like not making your data and code available for other researchers to check. They also include more subtle things like selectively reporting results, tweaking the analysis until you get a significant result (‘p-hacking’) and forming hypotheses after the results are known (‘HARKing’).

It is not only researchers who engage in QRPs. Journal editors often reject papers with null or weak findings because they believe such findings will be less impactful and less interesting to readers. This leads to an ‘exaggeration bias’, whereby the average effect size appears larger than it really is because null or weak findings are not represented in the published literature.

For example, suppose 10 studies are done, five of which find weak effects and five of which find strong effects. If only the latter five end up getting published, the published literature will exaggerate the effect size in question. Such variability in results is common when sample sizes are small (when studies are ‘under-powered’, to use the technical jargon).

Kimmel and colleagues obtained relevant statistics from 354 articles that were published in five top ecology journals between 2018 and 2020. In one part of their analysis, they looked at the distribution of t-statistics reported by the articles in their sample. The t-statistic is something you calculate, which tells you how likely your results would be under the assumption of no effect. If it is large, that indicates your results would be unlikely under this assumption and hence that there probably is an effect.

What’s important is that, by convention, t-statistics greater than 1.96 are deemed significant whereas those below 1.96 are deemed non-significant. So if researchers were selectively reporting their results, you’d expect to see a discontinuity in the distribution, with fewer than expected just below 1.96 and more than expected just above 1.96. And this is just what Kimmel and colleagues found!

Figure taken from ‘Empirical evidence of widespread exaggeration bias and selective reporting in ecology’

As you can see, there’s a dip in the distribution just below 1.96 and spike just above it. Which can only be explained by some combination of HARKing, p-hacking and selective reporting of results. In fact, Kimmel and colleagues found additional evidence for selective reporting. When they examined the distribution of t-statistics from articles’ appendices, there was no discontinuity around 1.96 – as shown below.

Figure taken from ‘Empirical evidence of widespread exaggeration bias and selective reporting in ecology’

Which means that researchers were accurately reporting results in their appendices but selectively reporting results in their main texts. (This make sense, given that articles are typically be judged on the basis of their main findings, rather than any supplementary results.)

Kimmel and colleagues also found evidence for exaggeration bias in the ecology literature. In addition, they found that although data were available for 80% of the articles in their sample, code was available for less than 20%. Which means that the vast majority of articles’ results could not be reproduced without rewriting the code from scratch.

“The published effect sizes in ecology journals,” the authors write, “exaggerate the importance of the ecological relationships that they aim to quantify”. This, in turn, “hinders the ability of empirical ecology to reliably contribute to science, policy, and management”.

The paper doesn’t identify which specific “ecological relationships” are most exaggerated, nor which areas of the field are most affected, but it does offer general grounds for scepticism. Keep this in mind the next time you’re reading the science section.

Tags: EcologyP-hackingThe Science

Donate

We depend on your donations to keep this site going. Please give what you can.

Donate Today

Comment on this Article

You’ll need to set up an account to comment if you don’t already have one. We ask for a minimum donation of £5 if you'd like to make a comment or post in our Forums.

Sign Up
Previous Post

Climate Activists Declare War on “Gas Guzzler” Cars… by Vandalising an Electric Tesla

Next Post

The Post Office Horizon Scandal Portends a Future of Digital Chaos

Subscribe
Login
Notify of
Please log in to comment

To join in with the discussion please make a donation to The Daily Sceptic.

Profanity and abuse will be removed and may lead to a permanent ban.

11 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments

NEWSLETTER

View today’s newsletter

To receive our latest news in the form of a daily email, enter your details here:

DONATE

PODCAST

Episode 36 of the Sceptic: Karl Williams on Starmer’s Phoney Immigration Crackdown, Dan Hitchens on the Assisted Suicide Bill and Tom Jones on Reform’s Local Council Challenge

by Richard Eldred
16 May 2025
0

LISTED ARTICLES

  • Most Read
  • Most Commented
  • Editor’s Picks

Chinese ‘Kill Switches’ Found in US Solar Farms

15 May 2025
by Will Jones

News Round-Up

16 May 2025
by Richard Eldred

Spy Agency Report on the Alleged “Extremism” of AfD Turns Out to Be So Stupid That it Destroys all Momentum for Banning the Party

16 May 2025
by Eugyppius

The Folly of Solar – a Dot on the Horizon Versus a Blight on the Land

16 May 2025
by Ben Pile

Civil Servants Threaten to Strike Over Trans Ban in Women’s Lavatories

16 May 2025
by Will Jones

The Folly of Solar – a Dot on the Horizon Versus a Blight on the Land

29

Civil Servants Threaten to Strike Over Trans Ban in Women’s Lavatories

26

Spy Agency Report on the Alleged “Extremism” of AfD Turns Out to Be So Stupid That it Destroys all Momentum for Banning the Party

19

News Round-Up

18

Chinese ‘Kill Switches’ Found in US Solar Farms

27

Trump’s Lesson in Remedial Education

16 May 2025
by Dr James Allan

Spy Agency Report on the Alleged “Extremism” of AfD Turns Out to Be So Stupid That it Destroys all Momentum for Banning the Party

16 May 2025
by Eugyppius

The Folly of Solar – a Dot on the Horizon Versus a Blight on the Land

16 May 2025
by Ben Pile

Renaud Camus on the Destruction of Western Education

15 May 2025
by Dr Nicholas Tate

‘Why Can’t We Talk About This?’

15 May 2025
by Richard Eldred

POSTS BY DATE

January 2024
M T W T F S S
1234567
891011121314
15161718192021
22232425262728
293031  
« Dec   Feb »

SOCIAL LINKS

Free Speech Union
  • Home
  • About us
  • Donate
  • Privacy Policy

Facebook

  • X

Instagram

RSS

Subscribe to our newsletter

© Skeptics Ltd.

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms below to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Articles
  • About
  • Archive
    • ARCHIVE
    • NEWS ROUND-UPS
  • Podcasts
  • Newsletter
  • Premium
  • Donate
  • Log In

© Skeptics Ltd.

wpDiscuz
You are going to send email to

Move Comment
Perfecty
Do you wish to receive notifications of new articles?
Notifications preferences