Does the psychology literature reflect reality? Thousands of studies have been published claiming this effects that and that affects this, and so on. But how many of these effects are actually real? A 2021 paper by Anne Scheel and colleagues looked at this question, and came to the disturbing conclusion that a large share of them might not be real. They might, in other words, be null effects masquerading as true (or ‘statistically significant’) effects.
The paper used a clever method. To understand it, you have to be aware of the two main sources of bias in the psychology literature (as well as the scientific literature more broadly).
The first is selective reporting, also known as the ‘file drawer problem’. Essentially, it’s much harder to get a null result published than it is to get a positive result published. Null results are regarded as ‘boring’ and ‘uninformative’ – not the sort of thing editors want filling up the pages of their vaunted journals. (This is despite the obvious fact that it’s often very useful to know when something isn’t true.) Not only that, but null results can ruffle people’s feathers. If a distinguished academic publishes a paper claiming that such-and-such is true, and then some other academic publishes his own paper showing the opposite is true, the first one might get rather perturbed. (Academics can be extremely petty, presumably because the stakes are so low.)
The second source of bias comes under the heading of questionable research practices or QRPs. These are things like: not making your data available for other researchers to check; tweaking your analysis until you get a positive result (‘p-hacking’); running many analyses but only reporting the ones that give positive results; and forming hypotheses after analysing the data (‘HARK-ing’).
In recent years, some journals and researchers have sought to address these two sources of bias through what are called ‘registered reports’. A registered report is an academic paper with two key features: it tests hypotheses that have been pre-registered via a time-stamped protocol posted online; it is submitted to a journal and accepted for publication before the data have been collected and analysed (i.e., entirely on the basis of the hypotheses and proposed methods). In virtue of these two features, registered reports are immune from both selective reporting and questionable research practices.
Returning to Scheel and colleagues, they compared the percentage of articles with a positive result in a sample of registered reports and a sample of standard reports (i.e., ordinary academic papers). To be specific, they checked whether the first hypothesis tested in each article was deemed by the authors to have been supported. Did they write something like, ‘Our first hypothesis was confirmed’, in other words.
What did they find? The results are shown below:
As you can see, the first hypothesis was supported in 96% of standard reports but only 44% of registered reports – a huge gap. Now, registered reports are more likely to constitute replications of previous articles, so they might be less likely to find support for their hypothesis for that reason alone. However, even when the authors excluded replication studies from both samples, there was still a massive difference of 46 percentage points.
This suggests that up to half the effects reported in psychology might not be real; one in every two studies could be making false claims. While Scheel and colleagues’ study has some limitations like any other, their findings suggest that selective reporting and questionable research practices are absolutely rampant. And in case you’re wondering, yes, they did pre-register their own hypotheses.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
Why don’t they write to their school administration and o their teachers, who are the ones pushing this lunacy?
Better still why don’t they have a word with the boys and girls in their schools pretending they are a different gender and tell them to stop acting up like mental patients.
The solution to these or any of our problems isn’t coming from the government. If we look to government to solve our problems we’re completely screwed.
School Administrators don’t make education policy. It is government who do that. Tyranny is a top down one. Administrators did not remove leather straps. Social Justice people did that and they are now all in government.
“Mixed Loos are ‘Intimidating, Humiliating and Downright Dangerous”
Well knocka me down with a feather! Who knew?
Another day in
world..who the hell would have thought girls would be having to beg and plead to have their own toilets? Who thinks this is a good idea…and why?
…..and where are the useless parents?? Why aren’t they challenging it?
If they can’t be bothered to sort it out and fight for their own children, I’m beginning not to care….
Every f***ing wrong thing in the world can’t only be a problem, and resolved, by people like us…and they are making me less likely to care by their own bloody lack of a spine….
The parents need to band together & make it clear to the headteacher, citing them by name, that as the man or woman (not person – in law it’s very important that little detail) if any girl is assaulted or comes to harm, then they, the individual named man or woman will be held individually liable.
Nothing like forcing accountability to make someone reconsider their action. At the moment, these folk making the decisions are hiding behind bureaucracy & have no personal skin in the game.
This agenda is being implemented because of fear of loss, use that fear of loss to force responsible behaviour.
..absolutely agree…I’m often bewildered by lack of parental interest…I felt the same over masking…class ‘bubbles’ ..and jabs for kids….
…I can’t imagine having young children at school and not organising or helping to organise strikes, non-attendance…anything to force the issue….
I’m seriously thinking that something has been put in jabs, or something that younger..as opposed to older people have taken to make them so blooming spineless and disinterested!!?
Grandparents, aunts, uncles could also serve notice to these named men & women. We the men & women who are awake to this evil agenda need to take a stand for same sex toilets, changing rooms, age inappropriate sex education & all the rest of it. The vast majority of our fellow men & women are still asleep, but just because they are, doesn’t mean that we can’t protect the children.
A lot of the current crop of parents have been well & truly indoctrinated within the
propagandaeducation system so don’t see any problem with what is going on.Good on you BB for posting this advice.
Five girls writing to their head teacher(s) would not have made it into the Mail.
The point is, men don’t like it either. In fact, no one likes it except for activists and they probably are in a position where they don’t have to use public toilets at all.
I have a suggestion for all girls being forced to be humiliated and discounted by their “educators”.
Vile as this sounds, teenage boys cannot actually cope with the unpleasant reality of periods. I suggest Ladies disgusting as it may sound you leave your used sanitary towels and tampons in the entrance over a line in your bathrooms. better still play the boys at their own game a dump the bins of used goods emptied in their toilets. you might want to consider doing it in your teachers toilets too. They might soon get the message.
I would add. if they want to play girl let them deal with the realty
Firm but fair.
Great idea. Reminds me of the tale of Hypatia of Alexandria, a neoplatonist philosopher of the 4th Century AD. In order to dissuade an unwanted suitor she waved her menstrual rags at him, saying something along the lines of “This is what you really love, my young man, for you do not love beauty for its own sake.” The historian Damascius reports that the chap was thoroughly traumatised and stopped his behaviour immediately!
As a parent who has tried to fight against this for my daughters, it’s practically impossible to turn the tide on this: school leadership and local authorities either stonewall or lie to you, Parent Councils are full of people more interested in ingratiating themselves with school leadership than properly representing families, other parents either prioritise virtue-signalling status, are utterly stupid or lack the courage to raise their heads above the parapet (see Covid for plenty of examples), and when you live under an incompetent devolved administration you get zero support from local elected (not by me) morons who would struggle to hold down managerial positions in McDonalds. In the end, we moved house and schools.
And if you want to take legal action you need deep pockets as LAs are very happy to tie up tax payers’ money in wasteful litigation and appeals.
How many know what “Scunnered” means?
Relevance?
Don’t worry there is nothing suspicious about me asking that question, and I am in no way critical of your sensible comment. The reason I ask is that I am from Scotland and I am wondering how many English and others know what “scunnered” means. —You have a nice day now.
Whever thought back 30 years ago that one day little girls would have to plead to government to not allow men into their toilets?
It’s a shame the girls writing to Sunak weren’t called Fatima, Ayesha, Maryam, Adya, and Leah …… he’d take more notice.
They could post this in the loos –
NHS conslutants will only try to sell this to do you if they think you’re a man. And even this only if you go to the doctor regularly enough for no particular reason that someone sees an easy sales opportunity here. I don’t. Hence, these people don’t molest me.
Incredible we’re having to have this conversation in the first instance. As I keep on saying the madness continues net zero, gene therapy treatments, masks, antisocial distancing, lockdowns the list goes on and on. Anyone twigged what’s going on yet?