Multireader evaluation of radiologist performance for COVID-19 detection on emergency department chest radiographs

• Authors: Judy Gichoya, Emory University; Priyanshu Sinha, Indiana University, Indianapolis; Melissa Davis, Emory University; Jeffrey W Dunkle, Indiana University, Indianapolis; Scott Hamlin, Emory University; Keith Herr, Emory University; Carrie Hoff, Emory University; Haley P Letter, University of Florida; Christopher McAdams, Emory University; Gregory D Puthoff, Wake Forest University; Kevin L Smith, Indiana Univ; Scott D Steenburg, Indiana Univ; Imon Banerjee, Emory University; Hari Trivedi, Emory University
• Journal Title: CLINICAL IMAGING
• Volume: Volume 82
• Publisher: ELSEVIER SCIENCE INC | 2021-11-16, Pages 77-82
• Type of Work: Article | Final Publisher PDF
• Publisher DOI: http://doi.org/10.1016/j.clinimag.2021.10.018
• Subjects: Health Sciences, Radiology
• Keywords: Science & Technology; Life Sciences & Biomedicine; Radiology, Nuclear Medicine & Medical Imaging; COVID-19; Coronavirus; Chest X-ray; CXR

Abstract

Background: Chest radiographs (CXR) are frequently used as a screening tool for patients with suspected COVID-19 infection pending reverse transcriptase polymerase chain reaction (RT-PCR) results, despite recommendations against this. We evaluated radiologist performance for COVID-19 diagnosis on CXR at the time of patient presentation in the Emergency Department (ED). Materials and methods: We extracted RT-PCR results, clinical history, and CXRs of all patients from a single institution between March and June 2020. 984 RT-PCR positive and 1043 RT-PCR negative radiographs were reviewed by 10 emergency radiologists from 4 academic centers. 100 cases were read by all radiologists and 1927 cases by 2 radiologists. Each radiologist chose the single best label per case: Normal, COVID-19, Other – Infectious, Other – Noninfectious, Non-diagnostic, and Endotracheal Tube. Cases labeled with endotracheal tube (246) or non-diagnostic (54) were excluded. Remaining cases were analyzed for label distribution, clinical history, and inter-reader agreement. Results: 1727 radiographs (732 RT-PCR positive, 995 RT-PCR negative) were included from 1594 patients (51.2% male, 48.8% female, age 59 ± 19 years). For 89 cases read by all readers, there was poor agreement for RT-PCR positive (Fleiss Score 0.36) and negative (Fleiss Score 0.46) exams. Agreement between two readers on 1638 cases was 54.2% (373/688) for RT-PCR positive cases and 71.4% (679/950) for negative cases. Agreement was highest for RT-PCR negative cases labeled as Normal (50.4%, n = 479). Reader performance did not improve with clinical history or time between CXR and RT-PCR result. Conclusion: At the time of presentation to the emergency department, emergency radiologist performance is non-specific for diagnosing COVID-19.