A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews

Omid Pournik; Emma Watts; Emma Richards; Kristien Boelaert; Neil Sharma; Saadullah Farooq Abbasi; Xuefei Ding; Leila Ghalichi; Theodoros N. Arvanitis

doi:10.3233/SHTI260255

A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews

Omid Pournik
, Emma Watts
, Emma Richards
, Kristien Boelaert
, Neil Sharma
, Saadullah Farooq Abbasi
, Xuefei Ding
, Leila Ghalichi
, Theodoros N. Arvanitis^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

24 Downloads (Pure)

Abstract

Systematic reviews are essential for evidence-based healthcare but remain highly resource-intensive, with most retrieved studies ultimately excluded after manual screening. This study developed and evaluated a hybrid expert-LLM workflow to reduce human workload while maintaining accuracy and transparency. Within the Thyroid Risk Stratification Tool (ThyRST) project, ChatGPT-5 was used to classify 14,858 records on thyroid nodule malignancy risk into thematic categories. Recurrent irrelevant concepts were refined through expert consensus involving clinicians and informaticians and embedded as exclusion rules in structured prompts. The model then labelled each abstract as INCLUDE, EXCLUDE, or MAYBE, producing outputs for audit and verification. A random sample of 100 records was independently reviewed by human assessors to evaluate performance. The workflow achieved 96% concordance (κ = 0.91) with human reviewers, with only one false exclusion, and reduced manual screening time by approximately 70%. These results demonstrate that a transparent Delphi-inspired expert-LLM can accurately and reproducibly automate early-stage evidence screening, providing substantial efficiency gains while preserving human oversight and methodological rigor. The approach offers a practical pathway toward the responsible integration of generative AI in systematic review methodology and digital health research.

Original language	English
Title of host publication	Opening the Personal Gate between Technology and Health Care
Subtitle of host publication	Proceedings of MIE 2026
Editors	Mauro Giacomini, Jaime Delgado, Theodoros N. Arvanitis, Elisavet Andrikopoulou, Arriel Benis, Gabriella Balestra, Riccardo Bellazzi, Parisis Gallos, Roberto Gatta, Daniele Roberto Giacobbe, Noemi Giordano, Maria Hägglund, Lars Lindsköld, Lenka Lhotska, Sara Marceglia, Enea Parimbelli, Lucia Sacchi, Paolo Soda, Lăcrămioara Stoicu-Tivadar, Pierangelo Veltri, Patrizia Vizza
Publisher	IOS Press
Pages	670-674
Number of pages	5
ISBN (Electronic)	9781643686615
DOIs	https://doi.org/10.3233/SHTI260255
Publication status	Published - 21 May 2026
Event	36th Medical Informatics Europe Conference, MIE 2026: Opening the Personal Gate between Technology and Health Care - Magazzini del Cotone, Genoa, Italy Duration: 25 May 2026 → 28 May 2026 https://mie2026.efmi.org/

Publication series

Name	Studies in Health Technology and Informatics
Publisher	IOS Press
Volume	336
ISSN (Print)	0926-9630
ISSN (Electronic)	1879-8365

Conference

Conference	36th Medical Informatics Europe Conference, MIE 2026
Abbreviated title	EFMI MIE2026
Country/Territory	Italy
City	Genoa
Period	25/05/26 → 28/05/26
Internet address	https://mie2026.efmi.org/

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Keywords

Humans
Workflow
Delphi Technique
Systematic Reviews as Topic
Thyroid Nodule/diagnosis
Evidence-Based Medicine
Thyroid Neoplasms/diagnosis
Risk Assessment/methods

ASJC Scopus subject areas

Biomedical Engineering
Health Informatics
Health Information Management

Access to Document

10.3233/SHTI260255Licence: Creative Commons: Attribution-NonCommercial (CC BY-NC)

PournikO2026HybridFinal published version, 211 KBLicence: Creative Commons: Attribution-NonCommercial (CC BY-NC)

Cite this

Pournik, O., Watts, E., Richards, E., Boelaert, K., Sharma, N., Abbasi, S. F., Ding, X., Ghalichi, L., & Arvanitis, T. N. (2026). A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews. In M. Giacomini, J. Delgado, T. N. Arvanitis, E. Andrikopoulou, A. Benis, G. Balestra, R. Bellazzi, P. Gallos, R. Gatta, D. R. Giacobbe, N. Giordano, M. Hägglund, L. Lindsköld, L. Lhotska, S. Marceglia, E. Parimbelli, L. Sacchi, P. Soda, L. Stoicu-Tivadar, P. Veltri, ... P. Vizza (Eds.), Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026 (pp. 670-674). (Studies in Health Technology and Informatics; Vol. 336). IOS Press. https://doi.org/10.3233/SHTI260255

Pournik, Omid ; Watts, Emma ; Richards, Emma et al. / A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews. Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. editor / Mauro Giacomini ; Jaime Delgado ; Theodoros N. Arvanitis ; Elisavet Andrikopoulou ; Arriel Benis ; Gabriella Balestra ; Riccardo Bellazzi ; Parisis Gallos ; Roberto Gatta ; Daniele Roberto Giacobbe ; Noemi Giordano ; Maria Hägglund ; Lars Lindsköld ; Lenka Lhotska ; Sara Marceglia ; Enea Parimbelli ; Lucia Sacchi ; Paolo Soda ; Lăcrămioara Stoicu-Tivadar ; Pierangelo Veltri ; Patrizia Vizza. IOS Press, 2026. pp. 670-674 (Studies in Health Technology and Informatics).

@inproceedings{d60d1efa71664171ba0aae8eb1ef0a80,

title = "A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews",

abstract = "Systematic reviews are essential for evidence-based healthcare but remain highly resource-intensive, with most retrieved studies ultimately excluded after manual screening. This study developed and evaluated a hybrid expert-LLM workflow to reduce human workload while maintaining accuracy and transparency. Within the Thyroid Risk Stratification Tool (ThyRST) project, ChatGPT-5 was used to classify 14,858 records on thyroid nodule malignancy risk into thematic categories. Recurrent irrelevant concepts were refined through expert consensus involving clinicians and informaticians and embedded as exclusion rules in structured prompts. The model then labelled each abstract as INCLUDE, EXCLUDE, or MAYBE, producing outputs for audit and verification. A random sample of 100 records was independently reviewed by human assessors to evaluate performance. The workflow achieved 96\% concordance (κ = 0.91) with human reviewers, with only one false exclusion, and reduced manual screening time by approximately 70\%. These results demonstrate that a transparent Delphi-inspired expert-LLM can accurately and reproducibly automate early-stage evidence screening, providing substantial efficiency gains while preserving human oversight and methodological rigor. The approach offers a practical pathway toward the responsible integration of generative AI in systematic review methodology and digital health research.",

keywords = "Humans, Workflow, Delphi Technique, Systematic Reviews as Topic, Thyroid Nodule/diagnosis, Evidence-Based Medicine, Thyroid Neoplasms/diagnosis, Risk Assessment/methods",

author = "Omid Pournik and Emma Watts and Emma Richards and Kristien Boelaert and Neil Sharma and Abbasi, \{Saadullah Farooq\} and Xuefei Ding and Leila Ghalichi and Arvanitis, \{Theodoros N.\}",

year = "2026",

month = may,

day = "21",

doi = "10.3233/SHTI260255",

language = "English",

series = "Studies in Health Technology and Informatics",

publisher = "IOS Press",

pages = "670--674",

editor = "Mauro Giacomini and Jaime Delgado and Arvanitis, \{Theodoros N.\} and Elisavet Andrikopoulou and Arriel Benis and Gabriella Balestra and Riccardo Bellazzi and Parisis Gallos and Roberto Gatta and Giacobbe, \{Daniele Roberto\} and Noemi Giordano and Maria H{\"a}gglund and Lars Lindsk{\"o}ld and Lenka Lhotska and Sara Marceglia and Enea Parimbelli and Lucia Sacchi and Paolo Soda and L{\u a}cr{\u a}mioara Stoicu-Tivadar and Pierangelo Veltri and Patrizia Vizza",

booktitle = "Opening the Personal Gate between Technology and Health Care",

address = "Netherlands",

note = "36th Medical Informatics Europe Conference, MIE 2026 : Opening the Personal Gate between Technology and Health Care, EFMI MIE2026 ; Conference date: 25-05-2026 Through 28-05-2026",

url = "https://mie2026.efmi.org/",

}

Pournik, O, Watts, E, Richards, E, Boelaert, K, Sharma, N, Abbasi, SF , Ding, X , Ghalichi, L & Arvanitis, TN 2026, A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews. in M Giacomini, J Delgado, TN Arvanitis, E Andrikopoulou, A Benis, G Balestra, R Bellazzi, P Gallos, R Gatta, DR Giacobbe, N Giordano, M Hägglund, L Lindsköld, L Lhotska, S Marceglia, E Parimbelli, L Sacchi, P Soda, L Stoicu-Tivadar, P Veltri & P Vizza (eds), Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. Studies in Health Technology and Informatics, vol. 336, IOS Press, pp. 670-674, 36th Medical Informatics Europe Conference, MIE 2026, Genoa, Italy, 25/05/26. https://doi.org/10.3233/SHTI260255

A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews. / Pournik, Omid; Watts, Emma; Richards, Emma et al.
Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. ed. / Mauro Giacomini; Jaime Delgado; Theodoros N. Arvanitis; Elisavet Andrikopoulou; Arriel Benis; Gabriella Balestra; Riccardo Bellazzi; Parisis Gallos; Roberto Gatta; Daniele Roberto Giacobbe; Noemi Giordano; Maria Hägglund; Lars Lindsköld; Lenka Lhotska; Sara Marceglia; Enea Parimbelli; Lucia Sacchi; Paolo Soda; Lăcrămioara Stoicu-Tivadar; Pierangelo Veltri; Patrizia Vizza. IOS Press, 2026. p. 670-674 (Studies in Health Technology and Informatics; Vol. 336).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews

AU - Pournik, Omid

AU - Watts, Emma

AU - Richards, Emma

AU - Boelaert, Kristien

AU - Sharma, Neil

AU - Abbasi, Saadullah Farooq

AU - Ding, Xuefei

AU - Ghalichi, Leila

AU - Arvanitis, Theodoros N.

PY - 2026/5/21

Y1 - 2026/5/21

N2 - Systematic reviews are essential for evidence-based healthcare but remain highly resource-intensive, with most retrieved studies ultimately excluded after manual screening. This study developed and evaluated a hybrid expert-LLM workflow to reduce human workload while maintaining accuracy and transparency. Within the Thyroid Risk Stratification Tool (ThyRST) project, ChatGPT-5 was used to classify 14,858 records on thyroid nodule malignancy risk into thematic categories. Recurrent irrelevant concepts were refined through expert consensus involving clinicians and informaticians and embedded as exclusion rules in structured prompts. The model then labelled each abstract as INCLUDE, EXCLUDE, or MAYBE, producing outputs for audit and verification. A random sample of 100 records was independently reviewed by human assessors to evaluate performance. The workflow achieved 96% concordance (κ = 0.91) with human reviewers, with only one false exclusion, and reduced manual screening time by approximately 70%. These results demonstrate that a transparent Delphi-inspired expert-LLM can accurately and reproducibly automate early-stage evidence screening, providing substantial efficiency gains while preserving human oversight and methodological rigor. The approach offers a practical pathway toward the responsible integration of generative AI in systematic review methodology and digital health research.

AB - Systematic reviews are essential for evidence-based healthcare but remain highly resource-intensive, with most retrieved studies ultimately excluded after manual screening. This study developed and evaluated a hybrid expert-LLM workflow to reduce human workload while maintaining accuracy and transparency. Within the Thyroid Risk Stratification Tool (ThyRST) project, ChatGPT-5 was used to classify 14,858 records on thyroid nodule malignancy risk into thematic categories. Recurrent irrelevant concepts were refined through expert consensus involving clinicians and informaticians and embedded as exclusion rules in structured prompts. The model then labelled each abstract as INCLUDE, EXCLUDE, or MAYBE, producing outputs for audit and verification. A random sample of 100 records was independently reviewed by human assessors to evaluate performance. The workflow achieved 96% concordance (κ = 0.91) with human reviewers, with only one false exclusion, and reduced manual screening time by approximately 70%. These results demonstrate that a transparent Delphi-inspired expert-LLM can accurately and reproducibly automate early-stage evidence screening, providing substantial efficiency gains while preserving human oversight and methodological rigor. The approach offers a practical pathway toward the responsible integration of generative AI in systematic review methodology and digital health research.

KW - Humans

KW - Workflow

KW - Delphi Technique

KW - Systematic Reviews as Topic

KW - Thyroid Nodule/diagnosis

KW - Evidence-Based Medicine

KW - Thyroid Neoplasms/diagnosis

KW - Risk Assessment/methods

U2 - 10.3233/SHTI260255

DO - 10.3233/SHTI260255

M3 - Conference contribution

C2 - 42174930

T3 - Studies in Health Technology and Informatics

SP - 670

EP - 674

BT - Opening the Personal Gate between Technology and Health Care

A2 - Giacomini, Mauro

A2 - Delgado, Jaime

A2 - Arvanitis, Theodoros N.

A2 - Andrikopoulou, Elisavet

A2 - Benis, Arriel

A2 - Balestra, Gabriella

A2 - Bellazzi, Riccardo

A2 - Gallos, Parisis

A2 - Gatta, Roberto

A2 - Giacobbe, Daniele Roberto

A2 - Giordano, Noemi

A2 - Hägglund, Maria

A2 - Lindsköld, Lars

A2 - Lhotska, Lenka

A2 - Marceglia, Sara

A2 - Parimbelli, Enea

A2 - Sacchi, Lucia

A2 - Soda, Paolo

A2 - Stoicu-Tivadar, Lăcrămioara

A2 - Veltri, Pierangelo

A2 - Vizza, Patrizia

PB - IOS Press

T2 - 36th Medical Informatics Europe Conference, MIE 2026

Y2 - 25 May 2026 through 28 May 2026

ER -

Pournik O, Watts E, Richards E, Boelaert K, Sharma N, Abbasi SF et al. A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews. In Giacomini M, Delgado J, Arvanitis TN, Andrikopoulou E, Benis A, Balestra G, Bellazzi R, Gallos P, Gatta R, Giacobbe DR, Giordano N, Hägglund M, Lindsköld L, Lhotska L, Marceglia S, Parimbelli E, Sacchi L, Soda P, Stoicu-Tivadar L, Veltri P, Vizza P, editors, Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. IOS Press. 2026. p. 670-674. (Studies in Health Technology and Informatics). doi: 10.3233/SHTI260255

A Hybrid Delphi-Inspired Expert-LLM Workflow for Efficient Evidence Screening in Systematic Reviews

Abstract

Publication series

Conference

UN SDGs

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this