Computer-aided diagnosis of renal obstruction: utility of log-linear modeling versus standard ROC and kappa analysis

• Type of Material: Article
• Authors: Amita Manatunga, Emory University; Jose N Binongo, Emory University; Andrew T Taylor Jr., Emory University
• Language: English
• Date: 2011-05-20
• Publisher: BioMed Central
• Publication Version: Final Published Version
• Copyright Statement: © 2011 Manatunga et al; licensee Springer.
• License: Creative Commons Attribution 2.0 Generic
• Final Published Version (URL): http://www.ejnmmires.com/content/1/1/5
• Title of Journal or Parent Work: EJNMMI Research
• ISSN: 2191-219X
• Volume: 1
• Issue: 5
• Start Page: 1
• End Page: 8
• Grant/Funding Information: This work was supported by a National Institute of Health grant, RO1-EB008838, funded from the National Institute of Biomedical Imaging and BioEngineering and from the National Institute of Diabetes and Digestive and Kidney Diseases and a URC grant from Emory University. We would also like to thank Eva V. Dubovsky, MD, PhD and Raghuveer Halkar, MD for serving as expert readers and Russell Folks, CNMT for his assistance in the acquisition and organization of the data.
• Research Categories: Health Sciences, Radiology; Biology, Biostatistics

Abstract

Background The accuracy of computer-aided diagnosis (CAD) software is best evaluated by comparison to a gold standard which represents the true status of disease. In many settings, however, knowledge of the true status of disease is not possible and accuracy is evaluated against the interpretations of an expert panel. Common statistical approaches to evaluate accuracy include receiver operating characteristic (ROC) and kappa analysis but both of these methods have significant limitations and cannot answer the question of equivalence: Is the CAD performance equivalent to that of an expert? The goal of this study is to show the strength of log-linear analysis over standard ROC and kappa statistics in evaluating the accuracy of computer-aided diagnosis of renal obstruction compared to the diagnosis provided by expert readers. Methods Log-linear modeling was utilized to analyze a previously published database that used ROC and kappa statistics to compare diuresis renography scan interpretations (non-obstructed, equivocal, or obstructed) generated by a renal expert system (RENEX) in 185 kidneys (95 patients) with the independent and consensus scan interpretations of three experts who were blinded to clinical information and prospectively and independently graded each kidney as obstructed, equivocal, or non-obstructed. Results Log-linear modeling showed that RENEX and the expert consensus had beyond-chance agreement in both non-obstructed and obstructed readings (both p < 0.0001). Moreover, pairwise agreement between experts and pairwise agreement between each expert and RENEX were not significantly different (p = 0.41, 0.95, 0.81 for the non-obstructed, equivocal, and obstructed categories, respectively). Similarly, the three-way agreement of the three experts and three-way agreement of two experts and RENEX was not significantly different for non-obstructed (p = 0.79) and obstructed (p = 0.49) categories. Conclusion Log-linear modeling showed that RENEX was equivalent to any expert in rating kidneys, particularly in the obstructed and non-obstructed categories. This conclusion, which could not be derived from the original ROC and kappa analysis, emphasizes and illustrates the role and importance of log-linear modeling in the absence of a gold standard. The log-linear analysis also provides additional evidence that RENEX has the potential to assist in the interpretation of diuresis renography studies.