Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study

Rose Allington; Nasim Mahmoodi; Omid Pournik; Theodoros N. Arvanitis

doi:10.3233/SHTI260322

Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study

Rose Allington
, Nasim Mahmoodi
, Omid Pournik
, Theodoros N. Arvanitis^*

^*Corresponding author for this work

Electronic, Electrical and Systems Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

15 Downloads (Pure)

Abstract

CPGs (Clinical Practice Guidelines) contain the best care practices for clinicians to use and are created in many formats. The development of LLMs (Large Language Models) has led to their use in extracting or adapting CPG content to improve the ease of access to information for clinicians. In this paper, we investigate four different LLMs' ability to extract clinical recommendations from guidelines and apply basic categories to each recommendation, with one test being performed with an example of extracted recommendations and one test without this example. Of the LLMs used, DeepSeek and Grok created the best outputs, extracting the most recommendations and extracting them most correctly, achieving >90% accuracy. While this model does show that LLMs show promise in knowledge extraction, this preliminary evaluation highlights both potential and limitations of LLMs in automating knowledge extraction from clinical guidelines.

Original language	English
Title of host publication	Opening the Personal Gate between Technology and Health Care
Subtitle of host publication	Proceedings of MIE 2026
Editors	Mauro Giacomini, Jaime Delgado, Theodoros N. Arvanitis, Elisavet Andrikopoulou, Arriel Benis, Gabriella Balestra, Riccardo Bellazzi, Parisis Gallos, Roberto Gatta, Daniele Roberto Giacobbe, Noemi Giordano, Maria Hägglund, Lars Lindsköld, Lenka Lhotska, Sara Marceglia, Enea Parimbelli, Lucia Sacchi, Paolo Soda, Lăcrămioara Stoicu-Tivadar, Pierangelo Veltri, Patrizia Vizza
Publisher	IOS Press
Pages	964-968
Number of pages	5
ISBN (Electronic)	9781643686615
DOIs	https://doi.org/10.3233/SHTI260322
Publication status	Published - 21 May 2026
Event	36th Medical Informatics Europe Conference, MIE 2026: Opening the Personal Gate between Technology and Health Care - Magazzini del Cotone, Genoa, Italy Duration: 25 May 2026 → 28 May 2026 https://mie2026.efmi.org/

Publication series

Name	Studies in Health Technology and Informatics
Publisher	IOS Press
Volume	336
ISSN (Print)	0926-9630
ISSN (Electronic)	1879-8365

Conference

Conference	36th Medical Informatics Europe Conference, MIE 2026
Abbreviated title	EFMI MIE2026
Country/Territory	Italy
City	Genoa
Period	25/05/26 → 28/05/26
Internet address	https://mie2026.efmi.org/

Keywords

AI
Clinical Practice Guidelines
Knowledge Extraction
Large Language Models

ASJC Scopus subject areas

Biomedical Engineering
Health Informatics
Health Information Management

Access to Document

10.3233/SHTI260322Licence: Creative Commons: Attribution-NonCommercial (CC BY-NC)

AllingtonR2026EvaluatingFinal published version, 208 KBLicence: Creative Commons: Attribution-NonCommercial (CC BY-NC)

Cite this

Allington, R., Mahmoodi, N., Pournik, O., & Arvanitis, T. N. (2026). Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study. In M. Giacomini, J. Delgado, T. N. Arvanitis, E. Andrikopoulou, A. Benis, G. Balestra, R. Bellazzi, P. Gallos, R. Gatta, D. R. Giacobbe, N. Giordano, M. Hägglund, L. Lindsköld, L. Lhotska, S. Marceglia, E. Parimbelli, L. Sacchi, P. Soda, L. Stoicu-Tivadar, P. Veltri, ... P. Vizza (Eds.), Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026 (pp. 964-968). (Studies in Health Technology and Informatics; Vol. 336). IOS Press. https://doi.org/10.3233/SHTI260322

Allington, Rose ; Mahmoodi, Nasim ; Pournik, Omid et al. / Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines : A Preliminary Study. Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. editor / Mauro Giacomini ; Jaime Delgado ; Theodoros N. Arvanitis ; Elisavet Andrikopoulou ; Arriel Benis ; Gabriella Balestra ; Riccardo Bellazzi ; Parisis Gallos ; Roberto Gatta ; Daniele Roberto Giacobbe ; Noemi Giordano ; Maria Hägglund ; Lars Lindsköld ; Lenka Lhotska ; Sara Marceglia ; Enea Parimbelli ; Lucia Sacchi ; Paolo Soda ; Lăcrămioara Stoicu-Tivadar ; Pierangelo Veltri ; Patrizia Vizza. IOS Press, 2026. pp. 964-968 (Studies in Health Technology and Informatics).

@inproceedings{54ed4d6e646b4d28bbcae545da75ba77,

title = "Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study",

abstract = "CPGs (Clinical Practice Guidelines) contain the best care practices for clinicians to use and are created in many formats. The development of LLMs (Large Language Models) has led to their use in extracting or adapting CPG content to improve the ease of access to information for clinicians. In this paper, we investigate four different LLMs' ability to extract clinical recommendations from guidelines and apply basic categories to each recommendation, with one test being performed with an example of extracted recommendations and one test without this example. Of the LLMs used, DeepSeek and Grok created the best outputs, extracting the most recommendations and extracting them most correctly, achieving >90\% accuracy. While this model does show that LLMs show promise in knowledge extraction, this preliminary evaluation highlights both potential and limitations of LLMs in automating knowledge extraction from clinical guidelines.",

keywords = "AI, Clinical Practice Guidelines, Knowledge Extraction, Large Language Models",

author = "Rose Allington and Nasim Mahmoodi and Omid Pournik and Arvanitis, \{Theodoros N.\}",

year = "2026",

month = may,

day = "21",

doi = "10.3233/SHTI260322",

language = "English",

series = "Studies in Health Technology and Informatics",

publisher = "IOS Press",

pages = "964--968",

editor = "Mauro Giacomini and Jaime Delgado and Arvanitis, \{Theodoros N.\} and Elisavet Andrikopoulou and Arriel Benis and Gabriella Balestra and Riccardo Bellazzi and Parisis Gallos and Roberto Gatta and Giacobbe, \{Daniele Roberto\} and Noemi Giordano and Maria H{\"a}gglund and Lars Lindsk{\"o}ld and Lenka Lhotska and Sara Marceglia and Enea Parimbelli and Lucia Sacchi and Paolo Soda and L{\u a}cr{\u a}mioara Stoicu-Tivadar and Pierangelo Veltri and Patrizia Vizza",

booktitle = "Opening the Personal Gate between Technology and Health Care",

address = "Netherlands",

note = "36th Medical Informatics Europe Conference, MIE 2026 : Opening the Personal Gate between Technology and Health Care, EFMI MIE2026 ; Conference date: 25-05-2026 Through 28-05-2026",

url = "https://mie2026.efmi.org/",

}

Allington, R, Mahmoodi, N , Pournik, O & Arvanitis, TN 2026, Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study. in M Giacomini, J Delgado, TN Arvanitis, E Andrikopoulou, A Benis, G Balestra, R Bellazzi, P Gallos, R Gatta, DR Giacobbe, N Giordano, M Hägglund, L Lindsköld, L Lhotska, S Marceglia, E Parimbelli, L Sacchi, P Soda, L Stoicu-Tivadar, P Veltri & P Vizza (eds), Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. Studies in Health Technology and Informatics, vol. 336, IOS Press, pp. 964-968, 36th Medical Informatics Europe Conference, MIE 2026, Genoa, Italy, 25/05/26. https://doi.org/10.3233/SHTI260322

Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study. / Allington, Rose; Mahmoodi, Nasim ; Pournik, Omid et al.
Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. ed. / Mauro Giacomini; Jaime Delgado; Theodoros N. Arvanitis; Elisavet Andrikopoulou; Arriel Benis; Gabriella Balestra; Riccardo Bellazzi; Parisis Gallos; Roberto Gatta; Daniele Roberto Giacobbe; Noemi Giordano; Maria Hägglund; Lars Lindsköld; Lenka Lhotska; Sara Marceglia; Enea Parimbelli; Lucia Sacchi; Paolo Soda; Lăcrămioara Stoicu-Tivadar; Pierangelo Veltri; Patrizia Vizza. IOS Press, 2026. p. 964-968 (Studies in Health Technology and Informatics; Vol. 336).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines

T2 - 36th Medical Informatics Europe Conference, MIE 2026

AU - Allington, Rose

AU - Mahmoodi, Nasim

AU - Pournik, Omid

AU - Arvanitis, Theodoros N.

PY - 2026/5/21

Y1 - 2026/5/21

N2 - CPGs (Clinical Practice Guidelines) contain the best care practices for clinicians to use and are created in many formats. The development of LLMs (Large Language Models) has led to their use in extracting or adapting CPG content to improve the ease of access to information for clinicians. In this paper, we investigate four different LLMs' ability to extract clinical recommendations from guidelines and apply basic categories to each recommendation, with one test being performed with an example of extracted recommendations and one test without this example. Of the LLMs used, DeepSeek and Grok created the best outputs, extracting the most recommendations and extracting them most correctly, achieving >90% accuracy. While this model does show that LLMs show promise in knowledge extraction, this preliminary evaluation highlights both potential and limitations of LLMs in automating knowledge extraction from clinical guidelines.

AB - CPGs (Clinical Practice Guidelines) contain the best care practices for clinicians to use and are created in many formats. The development of LLMs (Large Language Models) has led to their use in extracting or adapting CPG content to improve the ease of access to information for clinicians. In this paper, we investigate four different LLMs' ability to extract clinical recommendations from guidelines and apply basic categories to each recommendation, with one test being performed with an example of extracted recommendations and one test without this example. Of the LLMs used, DeepSeek and Grok created the best outputs, extracting the most recommendations and extracting them most correctly, achieving >90% accuracy. While this model does show that LLMs show promise in knowledge extraction, this preliminary evaluation highlights both potential and limitations of LLMs in automating knowledge extraction from clinical guidelines.

KW - AI

KW - Clinical Practice Guidelines

KW - Knowledge Extraction

KW - Large Language Models

U2 - 10.3233/SHTI260322

DO - 10.3233/SHTI260322

M3 - Conference contribution

C2 - 42174996

T3 - Studies in Health Technology and Informatics

SP - 964

EP - 968

BT - Opening the Personal Gate between Technology and Health Care

A2 - Giacomini, Mauro

A2 - Delgado, Jaime

A2 - Arvanitis, Theodoros N.

A2 - Andrikopoulou, Elisavet

A2 - Benis, Arriel

A2 - Balestra, Gabriella

A2 - Bellazzi, Riccardo

A2 - Gallos, Parisis

A2 - Gatta, Roberto

A2 - Giacobbe, Daniele Roberto

A2 - Giordano, Noemi

A2 - Hägglund, Maria

A2 - Lindsköld, Lars

A2 - Lhotska, Lenka

A2 - Marceglia, Sara

A2 - Parimbelli, Enea

A2 - Sacchi, Lucia

A2 - Soda, Paolo

A2 - Stoicu-Tivadar, Lăcrămioara

A2 - Veltri, Pierangelo

A2 - Vizza, Patrizia

PB - IOS Press

Y2 - 25 May 2026 through 28 May 2026

ER -

Allington R, Mahmoodi N , Pournik O , Arvanitis TN. Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study. In Giacomini M, Delgado J, Arvanitis TN, Andrikopoulou E, Benis A, Balestra G, Bellazzi R, Gallos P, Gatta R, Giacobbe DR, Giordano N, Hägglund M, Lindsköld L, Lhotska L, Marceglia S, Parimbelli E, Sacchi L, Soda P, Stoicu-Tivadar L, Veltri P, Vizza P, editors, Opening the Personal Gate between Technology and Health Care: Proceedings of MIE 2026. IOS Press. 2026. p. 964-968. (Studies in Health Technology and Informatics). doi: 10.3233/SHTI260322

Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary Study

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this