Projects per year
Abstract
Objectives
To determine the prevalence of poor interpretation practices, such as conflating evidence of absence with absence of evidence and over-emphasis of statistical non-significance in abstract conclusions, in a sample of randomized controlled trials (RCTs) with non-statistically significant primary outcomes published after the 2016 American Statistical Association statement on the interpretation of P-values.
Design and setting
Review of 50 two-arm individually randomized superiority trials with non-statistically significant results in four high impact journals published between 2017 and 2020, to determine the proportion that conclude evidence of no impact (thus, likely conflating evidence of absence with absence of evidence) or place emphasis on statistical non-significance (technically correct but arguably uninformative) in the abstract conclusion.
Results
Of the 50 RCTs with non-statistically significant results for primary outcomes, 28 (56%) of abstract were classified as concluding there was no difference between the two treatments; 19 (38%) placed an over-emphasis on statistical significance; only one acknowledged any uncertainty and the remaining 2 (4%) concluded that one treatment was more effective. Only four studies provided any justification for a finding of no difference, for example that the confidence interval gave no support to values of importance.
Conclusions
RCTs with non-statistically significant primary outcomes almost always present their conclusion in the abstract as evidence of no impact or ambiguously as “not statistically significant” without giving due attention to values supported by the confidence interval.
To determine the prevalence of poor interpretation practices, such as conflating evidence of absence with absence of evidence and over-emphasis of statistical non-significance in abstract conclusions, in a sample of randomized controlled trials (RCTs) with non-statistically significant primary outcomes published after the 2016 American Statistical Association statement on the interpretation of P-values.
Design and setting
Review of 50 two-arm individually randomized superiority trials with non-statistically significant results in four high impact journals published between 2017 and 2020, to determine the proportion that conclude evidence of no impact (thus, likely conflating evidence of absence with absence of evidence) or place emphasis on statistical non-significance (technically correct but arguably uninformative) in the abstract conclusion.
Results
Of the 50 RCTs with non-statistically significant results for primary outcomes, 28 (56%) of abstract were classified as concluding there was no difference between the two treatments; 19 (38%) placed an over-emphasis on statistical significance; only one acknowledged any uncertainty and the remaining 2 (4%) concluded that one treatment was more effective. Only four studies provided any justification for a finding of no difference, for example that the confidence interval gave no support to values of importance.
Conclusions
RCTs with non-statistically significant primary outcomes almost always present their conclusion in the abstract as evidence of no impact or ambiguously as “not statistically significant” without giving due attention to values supported by the confidence interval.
Original language | English |
---|---|
Pages (from-to) | 112-120 |
Number of pages | 9 |
Journal | Journal of Clinical Epidemiology |
Volume | 145 |
Early online date | 23 Jan 2022 |
DOIs | |
Publication status | E-pub ahead of print - 23 Jan 2022 |
Keywords
- Clinical importance
- Confidence intervals
- Conflating absence of evidence
- Misinterpretation
- Non-statistically significant
- P-values
- Randomised controlled trials
- Reporting
- Treatment effects
Fingerprint
Dive into the research topics of 'A review of high impact journals found that misinterpretation of non-statistically significant results from randomized trials was common'. Together they form a unique fingerprint.Projects
- 1 Finished
-
Improving the evidence base of healthcare policy interventions through the Stepped-wedge cluster randomised trial using mixed methodologies - OPEN FOR DISSEMINATION ONLY
Hemming, K. (Principal Investigator)
NIHR TRAINEES COORDINATING CENTRE
1/01/18 → 31/12/24
Project: Other Government Departments