Please use this identifier to cite or link to this item:
Title: Machine learning in clinical practice: Evaluation of an artificial intelligence tool after implementation.
Austin Authors: Akhlaghi, Hamed;Freeman, Sam;Vari, Cynthia;McKenna, Bede;Braitberg, George;Karro, Jonathan;Tahayori, Bahman
Affiliation: Department of Emergency Medicine, St Vincent's Hospital Melbourne, Melbourne, Victoria, Australia.;Department of Medical Education, The University of Melbourne, Melbourne, Victoria, Australia.;Faculty of Health, Deakin University, Melbourne, Victoria, Australia.
Department of Critical Care, The University of Melbourne, Melbourne, Victoria, Australia.
Florey Institute of Neuroscience and Mental Health, The University of Melbourne, Melbourne, Victoria, Australia.
Issue Date: 28-Sep-2023
Date: 2023
Publication information: Emergency Medicine Australasia : EMA 2023-09-28
Abstract: Artificial intelligence (AI) has gradually found its way into healthcare, and its future integration into clinical practice is inevitable. In the present study, we evaluate the accuracy of a novel AI algorithm designed to predict admission based on a triage note after clinical implementation. This is the first of such studies to investigate real-time AI performance in the emergency setting. The novel AI algorithm that predicts admission using a triage note was translated into clinical practice and integrated within St Vincent's Hospital Melbourne's electronic emergency patient management system. The data were collected from 1 January 2021 to 17 August 2022 to evaluate the diagnostic accuracy of the AI system after implementation. A total of 77 125 ED presentations were included. The live AI algorithm has a sensitivity of 73.1% (95% confidence interval 72.5-73.8), specificity of 74.3% (73.9-74.7), positive predictive value of 50% (49.6-50.4) and negative predictive value of 88.7% (88.5-89) with a total accuracy of 74% (73.7-74.3). The accuracy of the system was at the lowest for admission to psychiatric units (34%) and at the highest for gastroenterology and medical admission (84% and 80%, respectively). Our study showed the diagnostic evaluation of a real-time AI clinical decision-support tool became less accurate than the original. Although real-time sensitivity and specificity of the AI tool was still acceptable as a decision-support tool in the ED, we propose that continuous training and evaluation of AI-enabled clinical support tools in healthcare are conducted to ensure consistent accuracy and performance to prevent inadvertent consequences.
DOI: 10.1111/1742-6723.14325
ORCID: 0000-0002-3929-7265
Journal: Emergency Medicine Australasia : EMA
PubMed URL: 37771067
ISSN: 1742-6723
Type: Journal Article
Subjects: artificial intelligence
emergency department
machine learning
research translation
triage note
Appears in Collections:Journal articles

Show full item record

Page view(s)

checked on Dec 7, 2023

Google ScholarTM


Items in AHRO are protected by copyright, with all rights reserved, unless otherwise indicated.