Do Pre-registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?: Evidence from 15,992 Test Statistics and Suggestions for Improvement

Abel Brodeur; Nikolai  Cook; Jonathan Hartley; Anthony Heyes

Do Pre-registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias? Evidence from 15,992 Test Statistics and Suggestions for Improvement

Abel Brodeur, Nikolai Cook, Jonathan Hartley, Anthony Heyes

Economics

Research output: Working paper/Preprint › Discussion paper

97 Downloads (Pure)

Abstract

Pre-registration is regarded as an important contributor to research credibility. We investigate this by analyzing the pattern of test statistics from the universe of randomized controlled trials (RCT) studies published in 15 leading economics journals. We draw two conclusions: (a) Pre-registration frequently does not involve a pre-analysis plan (PAP), or sufficient detail to constrain meaningfully the actions and decisions of researchers after data is collected. Consistent with this, we find no evidence that pre-registration in itself reduces p-hacking and publication bias. (b) When pre-registration is accompanied by a PAP we find evidence consistent with both reduced p-hacking and publication bias.

Original language	English
Publisher	Institute for Replication (I4R)
Number of pages	72
Publication status	Published - 24 Jan 2024

Publication series

Name	I4R Discussion Paper
Publisher	Institute for Replication (I4R)
No.	101

Keywords

Research integrity
publication bias

Access to Document

do_pre_registration_I4RSubmitted manuscript, 3.22 MBLicence: None: All rights reserved

http://hdl.handle.net/10419/281136Licence: None: All rights reserved

Cite this

@techreport{44e193c67fbd436e98edbfa7cbaaaf09,

title = "Do Pre-registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?: Evidence from 15,992 Test Statistics and Suggestions for Improvement",

abstract = "Pre-registration is regarded as an important contributor to research credibility. We investigate this by analyzing the pattern of test statistics from the universe of randomized controlled trials (RCT) studies published in 15 leading economics journals. We draw two conclusions: (a) Pre-registration frequently does not involve a pre-analysis plan (PAP), or sufficient detail to constrain meaningfully the actions and decisions of researchers after data is collected. Consistent with this, we find no evidence that pre-registration in itself reduces p-hacking and publication bias. (b) When pre-registration is accompanied by a PAP we find evidence consistent with both reduced p-hacking and publication bias.",

keywords = "Research integrity, publication bias",

author = "Abel Brodeur and Nikolai Cook and Jonathan Hartley and Anthony Heyes",

year = "2024",

month = jan,

day = "24",

language = "English",

series = "I4R Discussion Paper",

publisher = "Institute for Replication (I4R)",

number = "101",

type = "WorkingPaper",

institution = "Institute for Replication (I4R)",

}

TY - UNPB

T1 - Do Pre-registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?

T2 - Evidence from 15,992 Test Statistics and Suggestions for Improvement

AU - Brodeur, Abel

AU - Cook, Nikolai

AU - Hartley, Jonathan

AU - Heyes, Anthony

PY - 2024/1/24

Y1 - 2024/1/24

N2 - Pre-registration is regarded as an important contributor to research credibility. We investigate this by analyzing the pattern of test statistics from the universe of randomized controlled trials (RCT) studies published in 15 leading economics journals. We draw two conclusions: (a) Pre-registration frequently does not involve a pre-analysis plan (PAP), or sufficient detail to constrain meaningfully the actions and decisions of researchers after data is collected. Consistent with this, we find no evidence that pre-registration in itself reduces p-hacking and publication bias. (b) When pre-registration is accompanied by a PAP we find evidence consistent with both reduced p-hacking and publication bias.

AB - Pre-registration is regarded as an important contributor to research credibility. We investigate this by analyzing the pattern of test statistics from the universe of randomized controlled trials (RCT) studies published in 15 leading economics journals. We draw two conclusions: (a) Pre-registration frequently does not involve a pre-analysis plan (PAP), or sufficient detail to constrain meaningfully the actions and decisions of researchers after data is collected. Consistent with this, we find no evidence that pre-registration in itself reduces p-hacking and publication bias. (b) When pre-registration is accompanied by a PAP we find evidence consistent with both reduced p-hacking and publication bias.

KW - Research integrity

KW - publication bias

M3 - Discussion paper

T3 - I4R Discussion Paper

BT - Do Pre-registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?

PB - Institute for Replication (I4R)

ER -