Extensive upfront validation and testing are needed prior to the clinical implementation of AI-based auto-segmentation tools

• Type of Material: Article
• Authors: Justin Roper, Emory University; Mu-Han Lin, University of Texas Southwestern; Yi Rong, Mayo Clinic Hospital
• Language: English
• Date: 2022-12-22
• Publisher: WILEY
• Publication Version: Final Published Version
• Copyright Statement: © 2022 The Authors. Journal of Applied Clinical Medical Physics published by Wiley Periodicals, LLC on behalf of The American Association of Physicists in Medicine.
• License: Creative Commons Attribution 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.1002/acm2.13873
• Title of Journal or Parent Work: JOURNAL OF APPLIED CLINICAL MEDICAL PHYSICS
• Volume: 24
• Issue: 1
• Start Page: e13873
• End Page: e13873
• Keywords: Radiology, Nuclear Medicine & Medical Imaging; Artificial Intelligence; Science & Technology; Life Sciences & Biomedicine; Model performance and QA; AI model prevalidation; Automatic segmentation
• Research Categories: Health Sciences, Radiology; Artificial Intelligence

Abstract

Artificial intelligence (AI) has emerged as a promising approach for automatic contouring in radiation therapy, compared to previous attempts, that is, image value thresholding, Atlas‐based, and so forth. Results from a 2017 AAPM Grand Challenge show that AI, specifically deep learning (DL), outperformed the previous gold standard model‐based methods for contouring thoracic anatomy on CT images. 1 In recent years, many clinics have started to adopt in‐house or commercial AI‐based autosegmentation tools in the clinic for various disease sites, as an attempt to save manual contouring time and speed up clinical workflow, as well as increasing contour quality consistency. Notice that for the majority of clinics, the group who develops or implements this tool (physicists) is often not the same group who routinely uses it (dosimetrists and physicians). Initial user feedback and satisfaction of these AI auto‐segmentation tools are highly heterogenous and inconsistent with the reported advantages and benefits in numerous publications. The dissatisfaction is mainly two folds: (1) the time saving on creating contours is diminished after having to manually correct for each contour, (2) the performance of the auto‐segmented contours varies greatly and sometimes can introduce risk factors. Amongst those early adopters of AI tools, the consensus is lacking regarding the required prior validation tests of AI auto‐segmentation models before releasing them for clinical use. For a successful clinical implementation, is extensive upfront quality assurance and validation important for ensuring the high performance of an AI model and identify pitfalls? Or is it more practical to recalibrate initial user expectation and establish an on‐going case monitoring strategy? Herein, we invited two experts with extensive experience in the clinical adoption of AI auto‐segmentation on this debate. Dr. Justin Roper is arguing for the proposition: “Extensive upfront validation and testing is needed prior to clinical implementation of AI‐based auto‐segmentation tools”, while Dr. Mu‐Han Lin is arguing against.