Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images

• Type of Material: Article
• Authors: Michael A. Marchetti, Memorial Sloan Kettering Cancer Center; Noel C.F. Codella, IBM Research Division; Stephen W. Dusza, Memorial Sloan Kettering Cancer Center; David Gutman, Emory University; Brian Helba, Kitware Inc; Aadi Kalloo, Memorial Sloan Kettering Cancer Center; Nabin Mishra, Stoecker & Associates; Cristina Carrera, University of Barcelona; M. Emre Celebi, University of Central Arkansas; Jennifer L. DeFazio, Memorial Sloan Kettering Cancer Center; Natalia Jaimes, Aurora Centro Especializado en Cancer del Piel; Ashfaq A. Marghoob, Memorial Sloan Kettering Cancer Center; Elizabeth Quigley, Memorial Sloan Kettering Cancer Center; Alon Scope, Memorial Sloan Kettering Cancer Center; Oriol Yelamos, Memorial Sloan Kettering Cancer Center; Allan C. Halpern, Memorial Sloan Kettering Cancer Center
• Language: English
• Date: 2018-02-01
• Publisher: Elsevier: 12 months
• Publication Version: Author Accepted Manuscript (After Peer Review)
• Copyright Statement: © 2017 American Academy of Dermatology, Inc.
• License: Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.1016/j.jaad.2017.08.016
• Title of Journal or Parent Work: Journal of The American Academy of Dermatology
• ISSN: 0190-9622
• Volume: 78
• Issue: 2
• Start Page: 270
• End Page: +
• Grant/Funding Information: This research was funded in part through the NIH/NCI Cancer Center Support Grant P30 CA008748.
• Keywords: computer algorithm; MULTICENTER; CUTANEOUS MELANOMA; skin cancer; computer vision; Dermatology; reader study; CANCER; PERFORMANCE; Life Sciences & Biomedicine; CLASSIFICATION; machine learning; dermatologist; Science & Technology; International Skin Imaging Collaboration; International Symposium on Biomedical Imaging; SYSTEM; melanoma
• Research Categories: Computer Science; Health Sciences, Medicine and Surgery

Abstract

Background: Computer vision may aid in melanoma detection. Objective: We sought to compare melanoma diagnostic accuracy of computer algorithms to dermatologists using dermoscopic images. Methods: We conducted a cross-sectional study using 100 randomly selected dermoscopic images (50 melanomas, 44 nevi, and 6 lentigines) from an international computer vision melanoma challenge dataset (n = 379), along with individual algorithm results from 25 teams. We used 5 methods (nonlearned and machine learning) to combine individual automated predictions into “fusion” algorithms. In a companion study, 8 dermatologists classified the lesions in the 100 images as either benign or malignant. Results: The average sensitivity and specificity of dermatologists in classification was 82% and 59%. At 82% sensitivity, dermatologist specificity was similar to the top challenge algorithm (59% vs. 62%, P =.68) but lower than the best-performing fusion algorithm (59% vs. 76%, P =.02). Receiver operating characteristic area of the top fusion algorithm was greater than the mean receiver operating characteristic area of dermatologists (0.86 vs. 0.71, P =.001). Limitations: The dataset lacked the full spectrum of skin lesions encountered in clinical practice, particularly banal lesions. Readers and algorithms were not provided clinical data (eg, age or lesion history/symptoms). Results obtained using our study design cannot be extrapolated to clinical practice. Conclusion: Deep learning computer vision systems classified melanoma dermoscopy images with accuracy that exceeded some but not all dermatologists.