Abstract
In-hospital cardiac arrest (IHCA) in the emergency department (ED) is not uncommon but often fatal. Using the machine learning (ML) approach, we sought to predict ED-based IHCA (EDCA) in patients presenting to the ED based on triage data. We retrieved 733,398 ED records from a tertiary teaching hospital over a 7 year period (Jan. 1, 2009–Dec. 31, 2015). We included only adult patients (≥ 18 y) and excluded cases presenting as out-of-hospital cardiac arrest. Primary outcome (EDCA) was identified via a resuscitation code. Patient demographics, triage data, and structured chief complaints (CCs), were extracted. Stratified split was used to divide the dataset into the training and testing cohort at a 3-to-1 ratio. Three supervised ML models were trained and performances were evaluated and compared to the National Early Warning Score 2 (NEWS2) and logistic regression (LR) model by the area under the receiver operating characteristic curve (AUC). We included 316,465 adult ED records for analysis. Of them, 636 (0.2%) developed EDCA. Of the constructed ML models, Random Forest outperformed the others with the best AUC result (0.931, 95% CI 0.911–0.949), followed by Gradient Boosting (0.930, 95% CI 0.909–0.948) and Extra Trees classifier (0.915, 95% CI 0.892–0.936). Although the differences between each of ML models and LR (AUC: 0.905, 95% CI 0.882–0.926) were not significant, all constructed ML models performed significantly better than using the NEWS2 scoring system (AUC 0.678, 95% CI 0.635–0.722). Our ML models showed excellent discriminatory performance to identify EDCA based only on the triage information. This ML approach has the potential to reduce unexpected resuscitation events if successfully implemented in the ED information system.
Similar content being viewed by others
Data Availability
Not available due to institutional restrictions.
Abbreviations
- IHCA:
-
In-hospital cardiac arrest
- OHCA:
-
Out-of-hospital cardiac arrest
- ED:
-
Emergency department
- EMR:
-
Electronic medical record
- EWS:
-
Early warning score
- EDCA:
-
ED-based IHCA
- ML:
-
Machine learning
- NEWS:
-
National early warning score
- CC:
-
Chief complaint
- iMD:
-
Integrated medical database
- NTUH:
-
National Taiwan University Hospital
- TTAS:
-
Taiwan triage and acuity scale
- EMS:
-
Emergency medical services
- GCS:
-
Glasgow coma scale
- SpO2:
-
Oxygen saturation
- SD:
-
Standard deviation
- RF:
-
Random forest
- GB:
-
Gradient boosting
- ET:
-
Extra trees
- AUC:
-
Area under the receiver operating characteristic curve
- AUPRC:
-
Area under the precision recall curve
- NPV:
-
Negative predictive value
- PPV:
-
Positive predictive value
- SMOTE:
-
Synthetic minority oversampling technique
- ESI:
-
Emergency severity index
- CTAS:
-
Canadian emergency department triage and acuity scale
- MTS:
-
Manchester triage system
References
Andersen LW, Holmberg MJ, Berg KM, Donnino MW, Granfeldt A (2019) In-hospital cardiac arrest: a review. JAMA 321(12):1200–1210. https://doi.org/10.1001/jama.2019.1696
Cummins RO, Chamberlain D, Hazinski MF, Nadkarni V, Kloeck W, Kramer E, Becker L et al (1997) Recommended guidelines for reviewing, reporting, and conducting research on in-hospital resuscitation: the in-hospital “Utstein style’. Am Heart Assoc Ann Emerg Med 29:650–679. https://doi.org/10.1016/s0196-0644(97)70256-7
Jacobs I, Nadkarni V, Bahr J, Berg RA, Billi JE, Bossaert L et al (2004) Cardiac arrest and cardiopulmonary resuscitation outcome reports: update and simplification of the Utstein templates for resuscitation registries: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian Resuscitation Council, New Zealand Resuscitation Council, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Councils of Southern Africa). Circulation 110:3385–3397. https://doi.org/10.1161/01.CIR.0000147236.85306.15
Yan S, Gan Y, Jiang N, Wang R, Chen Y, Luo Z et al (2020) The global survival rate among adult out-of-hospital cardiac arrest patients who received cardiopulmonary resuscitation: a systematic review and meta-analysis. Crit Care 24:61. https://doi.org/10.1186/s13054-020-2773-2
Kumar G, Nanchal R (2013) Trends in survival after in-hospital cardiac arrest. N Engl J Med 14(368):680. https://doi.org/10.1056/NEJMc1215155
Camargo CA Jr, Tsai CL, Sullivan AF, Cleary PD, Gordon JA, Guadagnoli E et al (2012) Safety climate and medical errors in 62 US emergency departments. Ann Emerg Med 60:555-563.e20. https://doi.org/10.1016/j.annemergmed.2012.02.018
Horwitz LI, Green J, Bradley EH (2010) US emergency department performance on wait time and length of visit. Ann Emerg Med 55:133–141. https://doi.org/10.1016/j.annemergmed.2009.07.023
Churpek MM, Yuen TC, Huber MT, Park SY, Hall JB, Edelson DP (2012) Predicting cardiac arrest on the wards: a nested case-control study. Chest 141:1170–1176. https://doi.org/10.1378/chest.11-1301
Gerry S, Bonnici T, Birks J, Kirtley S, Virdee PS, Watkinson PJ et al (2020) Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ 369:m1501. https://doi.org/10.1136/bmj.m1501
Goecks J, Jalili V, Heiser LM, Gray JW (2020) How machine learning will transform biomedicine. Cell 181:92–101. https://doi.org/10.1016/j.cell.2020.03.022
Rajkomar A, Dean J, Kohane I (2019) Machine Learning in Medicine. Reply N Engl J Med 27(380):2589–2590. https://doi.org/10.1056/NEJMc1906060
Chang HK, Wu CT, Liu JH, Lim WS, Wang HC, Chiu SI et al (2019) Early Detecting In-Hospital Cardiac Arrest Based on Machine Learning on Imbalanced Data, 2019 IEEE International Conference on Healthcare Informatics (ICHI), pp. 1–10. https://doi.org/10.1109/ICHI.2019.8904504.
Royal College of Physicians National Early Warning Score (NEWS): Standardising the assessment of acute illness severity in the NHS. Report of a working party. London: RCP, 2012 https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news-2. Accessed 3 May 2022
Ng CJ, Yen ZS, Tsai JC, Chen LC, Lin SJ, Sang YY et al (2011) Validation of the Taiwan triage and acuity scale: a new computerised five-level triage system. Emerg Med J 28:1026–1031. https://doi.org/10.1136/emj.2010.094185
Nolan JP, Berg RA, Andersen LW, Bhanji F, Chan PS, Donnino MW et al (2019) Cardiac arrest and cardiopulmonary resuscitation outcome reports: update of the utstein resuscitation registry template for in-hospital cardiac arrest: a consensus report from a task force of the international liaison committee on resuscitation (American Heart Association, European Resuscitation Council, Australian and New Zealand Council on Resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Council of Southern Africa, Resuscitation Council of Asia). Resuscitation 144:166–177. https://doi.org/10.1016/j.resuscitation.2019.08.021
Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Art Intell Res 16:321–357. https://doi.org/10.1613/jair.953
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O et al (2011) Scikit-learn: machine learning in python. JMLR 12:2825–2830
S. Džeroski P, Panov B, Ženko, (2009) Ensemble methods in machine learning. In: Meyer T (ed) Encyclopedia of Complexity and Systems Science. Springer, New York, pp 5317–5325
Trevethan R (2017) Sensitivity, specificity, and predictive values: foundations, pliabilities, and pitfalls in research and practice. Front Public Health 5:307. https://doi.org/10.3389/fpubh.2017.00307
Ranganathan P, Aggarwal R (2018) Common pitfalls in statistical analysis: understanding the properties of diagnostic tests-part 1. Perspect Clin Res 9:40–43. https://doi.org/10.4103/picr.PICR_170_17
Peberdy MA, Kaye W, Ornato JP, Larkin GL, Nadkarni V, Mancini ME et al (2003) Cardiopulmonary resuscitation of adults in the hospital: a report of 14720 cardiac arrests from the national registry of cardiopulmonary resuscitation. Resuscitation 58:297–308. https://doi.org/10.1016/s0300-9572(03)00215-6
Shih CL, Lu TC, Jerng JS, Lin CC, Liu YP, Chen WJ et al (2007) A web-based Utstein style registry system of in-hospital cardiopulmonary resuscitation in Taiwan. Resuscitation 72:394–403. https://doi.org/10.1016/j.resuscitation.2006.07.020
Høybye M, Stankovic N, Holmberg M, Christensen HC, Granfeldt A, Andersen LW (2021) In-hospital vs. out-of-hospital cardiac arrest: patient characteristics and survival. Resuscitation 158:157–165. https://doi.org/10.1016/j.resuscitation.2020.11.016
Christ M, Grossmann F, Winter D, Bingisser R, Platz E (2010) Modern triage in the emergency department. Dtsch Arztebl Int 107:892–898. https://doi.org/10.3238/arztebl.2010.0892
van der Wulp I, Schrijvers AJ, van Stel HF (2009) Predicting admission and mortality with the emergency severity index and the manchester triage system: a retrospective observational study. Emerg Med J 26:506–509. https://doi.org/10.1136/emj.2008.063768
Martins HM, Cuña LM, Freitas P (2009) Is manchester (MTS) more than a triage system? A study of its association with mortality and admission to a large Portuguese hospital. Emerg Med J 26:183–186. https://doi.org/10.1136/emj.2008.060780
Hinson JS, Martinez DA, Schmitz PSK, Toerper M, Radu D, Scheulen J et al (2018) Accuracy of emergency department triage using the emergency severity index and independent predictors of under-triage and over-triage in Brazil: a retrospective cohort analysis. Int J Emerg Med 11:3. https://doi.org/10.1186/s12245-017-0161-8
Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D (2016) An electronic emergency triage system to improve patient distribution by critical outcomes. J Emerg Med 50:910–918. https://doi.org/10.1016/j.jemermed.2016.02.026
Mitchell OJL, Edelson DP, Abella BS (2020) Predicting cardiac arrest in the emergency department. J Am Coll Emerg Physicians Open 1:321–326. https://doi.org/10.1002/emp2.12015
Green M, Lander H, Snyder A, Hudson P, Churpek M, Edelson D (2018) Comparison of the between the flags calling criteria to the mews, news and the electronic cardiac arrest risk triage (eCART) score for the identification of deteriorating ward patients. Resuscitation 123:86–91. https://doi.org/10.1016/j.resuscitation.2017.10.028
Ong ME, Lee Ng CH, Goh K, Liu N, Koh ZX, Shahidah N et al (2012) Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score. Crit Care 16:R108. https://doi.org/10.1186/cc11396
Chae M, Han S, Gil H, Cho N, Lee H (2021) Prediction of in-hospital cardiac arrest using shallow and deep learning. Diagnostics (Basel) 11(7):1255. https://doi.org/10.3390/diagnostics11071255
Kwon JM, Lee Y, Lee Y, Lee S, Park J (2018) An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc 7:e008678. https://doi.org/10.1161/JAHA.118.008678
Kohl M (2012) Performance Measures in Binary Classification. Int J Stat Med Res 1:79–81. https://doi.org/10.6000/1929-6029.2012.01.01.08
Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10(3):e0118432. https://doi.org/10.1371/journal.pone.0118432
Alghamdi M, Al-Mallah M, Keteyian S, Brawner C, Ehrman J, Sakr S (2017) Predicting diabetes mellitus using SMOTE and ensemble machine learning approach: The Henry Ford ExercIse Testing (FIT) project. PLoS ONE 12:e0179805. https://doi.org/10.1371/journal.pone.0179805
Fotouhi S, Asadi S, Kattan MW (2019) A comprehensive data level analysis for cancer diagnosis on imbalanced data. J Biomed Inform 90:103089. https://doi.org/10.1016/j.jbi.2018.12.003
Lee YW, Choi JW, Shin EH (2021) Machine learning model for predicting malaria using clinical information. Comput Biol Med 129:104151. https://doi.org/10.1016/j.compbiomed.2020.104151
Sung CW, Lu TC, Wang CH, Chou EH, Ko CH, Huang CH et al (2022) In-hospital cardiac arrest in united states emergency departments, 2010–2018. Front Cardiovasc Med 9:874461. https://doi.org/10.3389/fcvm.2022.874461
Acknowledgements
The authors would like to express their thanks to the staff of Department of Medical Research for providing clinical data from National Taiwan University Hospital-integrated Medical Database (NTUH-iMD).
Funding
This work was supported by the Ministry of Science and Technology Taiwan [111-2634-F-002 -015]; and the National Taiwan University Hospital [105-N3102]. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. All of the authors report no conflict of interest.
Author information
Authors and Affiliations
Contributions
Conception and design of the study: TCL, CHW, CLT. Acquisition of data: TCL, CLT, CHH. Analysis and interpretation of data: TCL, FYC. Drafting the article: TCL, CHW. Revising it critically for important intellectual content: JTS, EHC, MHMM, CCF. Final approval of the version to be submitted: All authors contributed to the final version. CLT takes responsibility for the paper as a whole.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethics approval
This study was approved by the institutional review board of the National Taiwan University Hospital (201606072RINA) and waived the requirement for informed consent.
Consent for publication
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lu, TC., Wang, CH., Chou, FY. et al. Machine learning to predict in-hospital cardiac arrest from patients presenting to the emergency department. Intern Emerg Med 18, 595–605 (2023). https://doi.org/10.1007/s11739-022-03143-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11739-022-03143-1