Developing a framework for assessing the applicability of the target condition in diagnostic research

Objective: Assessment of the applicability of primary studies is an essential but often challenging aspect of systematic reviews of diagnostic test accuracy studies (DTA reviews). We explored review authors’ applicability assessments for the QUADAS-2 reference standard domain within Cochrane DTA reviews. We highlight applicability concerns, identify potential issues with assessment, and develop a framework for assessing the applicability of the target condition as defined by the reference standard.

Study Design and Setting: Methodological review. DTA reviews in the Cochrane Library that used QUADAS-2 and judged applicability for the reference standard domain as “high concern” for at least one study, were eligible. One reviewer extracted the rationale for the “high concern”, this was checked by a second reviewer. Two reviewers categorized the rationale inductively into themes, a third reviewer verified these. Discussions regarding the extracted information informed framework development.

Results: We identified 50 eligible reviews. Five themes emerged: study uses different reference standard threshold to define the target condition (6 reviews), misclassification by the reference standard in the study such that the target condition in the study does not match the review question (11 reviews), reference standard could not be applied to all participants resulting in a different target condition (5 reviews), misunderstanding QUADAS-2 applicability (7 reviews), and insufficient information (21 reviews).
Our framework for researchers outlines four potential applicability concerns for the assessment of the target condition as defined by the reference standard: different sub-categories of the target condition, different threshold used to define the target condition, reference standard not applied to full study group, and misclassification of the target condition by the reference standard.

Conclusion: Clear sources of applicability concerns are identifiable, but several Cochrane review authors struggle to adequately identify and report them. We have developed an applicability framework to guide review authors in their assessment of applicability concerns for the QUADAS reference standard domain.