Classification of 12-lead ECGs: the PhysioNet/Computing in Cardiology Challenge 2020

• Type of Material: Article
• Authors: Erick A Perez Alday, Emory University; Annie Gu, Emory University; Amit Shah, Emory University; Chad Robichaux, Emory University; An-Kwok Ian Wong, Emory University; Chengyu Liu, Southeast University; Feifei Liu, Shandong Jianzhu University; Ali Bahrami Rad, Emory University; Andoni Elola, Emory University; Salman Seyedi, Emory University; Qiao Li, Emory University; Ashish Sharma, Emory University; Gari Clifford, Emory University; Matthew Reyna, Emory University
• Language: English
• Date: 2020-12-01
• Publisher: IOP Publishing Ltd
• Publication Version: Author Accepted Manuscript (After Peer Review)
• Copyright Statement: © 2020 Institute of Physics and Engineering in Medicine
• Final Published Version (URL): http://dx.doi.org/10.1088/1361-6579/abc960
• Title of Journal or Parent Work: PHYSIOLOGICAL MEASUREMENT
• Volume: 41
• Issue: 12
• Start Page: 124003
• End Page: 124003
• Grant/Funding Information: This research is supported by the National Institute of General Medical Sciences (NIGMS) and the National Institute of Biomedical Imaging and Bioengineering (NIBIB) under NIH Grant Numbers 2R01GM10 4987-09 and R01EB030362 respectively, the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002378, as well as the Gordon and Betty Moore Foundation, MathWorks, and AliveCor, Inc. under unrestricted gifts. Google also donated cloud compute credits for Challenge teams.
• Keywords: signal processing; Biophysics; Engineering, Biomedical; RHYTHM; Physiology; ELECTROCARDIOGRAM; Engineering; Technology; reproducibility; Life Sciences & Biomedicine; generalizability; PhysioNet; electrocardiogram; competition; Science & Technology

Abstract

Objective: Vast 12-lead ECGs repositories provide opportunities to develop new machine learning approaches for creating accurate and automatic diagnostic systems for cardiac abnormalities. However, most 12-lead ECG classification studies are trained, tested, or developed in single, small, or relatively homogeneous datasets. In addition, most algorithms focus on identifying small numbers of cardiac arrhythmias that do not represent the complexity and difficulty of ECG interpretation. This work addresses these issues by providing a standard, multi-institutional database and a novel scoring metric through a public competition: the PhysioNet/Computing in Cardiology Challenge 2020. Approach: A total of 66 361 12-lead ECG recordings were sourced from six hospital systems from four countries across three continents; 43 101 recordings were posted publicly with a focus on 27 diagnoses. For the first time in a public competition, we required teams to publish open-source code for both training and testing their algorithms, ensuring full scientific reproducibility. Main results: A total of 217 teams submitted 1395 algorithms during the Challenge, representing a diversity of approaches for identifying cardiac abnormalities from both academia and industry. As with previous Challenges, high-performing algorithms exhibited significant drops (10%) in performance on the hidden test data. Significance: Data from diverse institutions allowed us to assess algorithmic generalizability. A novel evaluation metric considered different misclassification errors for different cardiac abnormalities, capturing the outcomes and risks of different diagnoses. Requiring both trained models and code for training models improved the generalizability of submissions, setting a new bar in reproducibility for public data science competitions.