Generalization in Clinical Prediction Models: The Blessing and Curse of Measurement Indicator Variables.

• Authors: Joseph Futoma, Harvard University; Morgan Simons, NYU Langone Health; Finale Doshi-Velez, Harvard University; Rishikesan Kamaleswaran, Emory University
• Journal Title: Crit Care Explor
• Volume: Volume 3, Number 7
• Publisher: Wolters Kluwer Health, Inc. | 2021-07, Pages e0453-e0453
• Type of Work: Article | Final Publisher PDF
• Publisher DOI: http://doi.org/10.1097/CCE.0000000000000453
• Subjects: Health Sciences, Medicine and Surgery
• Keywords: decision support tools; external validity; generalizability; machine learning; statistical modeling; vasopressor therapy

Abstract

OBJECTIVE: Specific factors affecting generalizability of clinical prediction models are poorly understood. Our main objective was to investigate how measurement indicator variables affect external validity in clinical prediction models for predicting onset of vasopressor therapy. DESIGN: We fit logistic regressions on retrospective cohorts to predict vasopressor onset using two classes of variables: seemingly objective clinical variables (vital signs and laboratory measurements) and more subjective variables denoting recency of measurements. SETTING: Three cohorts from two tertiary-care academic hospitals in geographically distinct regions, spanning general inpatient and critical care settings. PATIENTS: Each cohort consisted of adult patients (age greater than or equal to 18 yr at time of hospitalization), with lengths of stay between 6 and 600 hours, and who did not receive vasopressors in the first 6 hours of hospitalization or ICU admission. Models were developed on each of the three derivation cohorts and validated internally on the derivation cohort and externally on the other two cohorts. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The prevalence of vasopressors was 0.9% in the general inpatient cohort and 12.4% and 11.5% in the two critical care cohorts. Models utilizing both classes of variables performed the best in-sample, with C-statistics for predicting vasopressor onset in 4 hours of 0.862 (95% CI, 0.844-0.879), 0.822 (95% CI, 0.793-0.852), and 0.889 (95% CI, 0.880-0.898). Models solely using the subjective variables denoting measurement recency had poor external validity. However, these practice-driven variables helped adjust for differences between the two hospitals and led to more generalizable models using clinical variables. CONCLUSIONS: We developed and externally validated models for predicting the onset of vasopressors. We found that practice-specific features denoting measurement recency improved local performance and also led to more generalizable models if they are adjusted for during model development but discarded at validation. The role of practice-specific features such as measurement indicators in clinical prediction modeling should be carefully considered if the goal is to develop generalizable models.