A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common

Karla Hemming, Iqra Javid, Monica Taljaard

Research output: Contribution to journalArticlepeer-review

Abstract

Objectives
To determine the prevalence of poor interpretation practices, such as conflating evidence of absence with absence of evidence and over-emphasis of statistical non-significance in abstract conclusions, in a sample of randomized controlled trials (RCTs) with non-statistically significant primary outcomes published after the 2016 American Statistical Association statement on the interpretation of P-values.
Design and setting
Review of 50 two-arm individually randomized superiority trials with non-statistically significant results in four high impact journals published between 2017 and 2020, to determine the proportion that conclude evidence of no impact (thus, likely conflating evidence of absence with absence of evidence) or place emphasis on statistical non-significance (technically correct but arguably uninformative) in the abstract conclusion.
Results
Of the 50 RCTs with non-statistically significant results for primary outcomes, 28 (56%) of abstract were classified as concluding there was no difference between the two treatments; 19 (38%) placed an over-emphasis on statistical significance; only one acknowledged any uncertainty and the remaining 2 (4%) concluded that one treatment was more effective. Only four studies provided any justification for a finding of no difference, for example that the confidence interval gave no support to values of importance.
Conclusions
RCTs with non-statistically significant primary outcomes almost always present their conclusion in the abstract as evidence of no impact or ambiguously as “not statistically significant” without giving due attention to values supported by the confidence interval.
Original languageEnglish
Pages (from-to)112-120
Number of pages9
JournalJournal of Clinical Epidemiology
Volume145
Early online date23 Jan 2022
DOIs
Publication statusE-pub ahead of print - 23 Jan 2022

Keywords

  • Non-statistically significant
  • Conflating absence of evidence
  • Misinterpretation
  • Randomised controlled trials
  • Reporting
  • Treatment effects
  • Confidence intervals
  • P-values
  • Clinical importance

Fingerprint

Dive into the research topics of 'A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common'. Together they form a unique fingerprint.

Cite this