Rating the quality of a body of evidence on the effectiveness of health and social interventions: a systematic review and mapping of evidence domains

Introduction: Rating the quality of a body of evidence is an increasingly common
component of research syntheses on intervention effectiveness. This study
sought to identify and examine existing systems for rating the quality of a body
of evidence on the effectiveness of health and social interventions.

Methods: We used a multi-component search strategy to search for full‐length
reports of systems for rating the quality of a body of evidence on the effectiveness of health and social interventions published in English from 1995 onward. Two independent reviewers extracted data from each eligible system on the evidence domains included, as well as the development and dissemination processes for each system.

Results: Seventeen systems met our eligibility criteria. Across systems, we
identified 13 discrete evidence domains: study design, study execution, consistency, measures of precision, directness, publication bias, magnitude of effect, dose‐response, plausible confounding, analogy, robustness, applicability, and coherence. We found little reporting of rigorous procedures in the development and dissemination of evidence rating systems.

Conclusion: We identified 17 systems for rating the quality of a body of evidence on intervention effectiveness across health and social policy. Existing systems vary greatly in the domains they include and how they operationalize domains, and most have important limitations in their development and dissemination. The construct of the quality of the body of evidence was defined in a few systems largely extending the Grading of Recommendations Assessment, Development, and Evaluation approach. Grading of Recommendations Assessment, Development, and Evaluation was found to be unique in its comprehensive guidance, rigorous development, and dissemination strategy.