Improved Small-Sample Estimation of Nonlinear Cross-Validated Prediction Metrics

• Type of Material: Article
• Authors: David Benkeser, Emory University; Maya Petersen, University of California Berkeley; Mark J van der Laan, University of California Berkeley
• Language: English
• Date: 2019-10-16
• Publisher: AMER STATISTICAL ASSOC
• Publication Version: Author Accepted Manuscript (After Peer Review)
• Copyright Statement: © 2018 Taylor & Francis
• License: Creative Commons Attribution-NonCommercial 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.1080/01621459.2019.1668794
• Title of Journal or Parent Work: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• Volume: 115
• Issue: 532
• Start Page: 1917
• End Page: 1932
• Keywords: Targeted minimum loss-based estimation; Estimating equation; Cross-validation; Science & Technology; Area under the receiver operating characteristic curve; Machine learning; Statistics & Probability; Prediction; Physical Sciences; Mathematics; CLASSIFICATION; MODELS

Abstract

When predicting an outcome is the scientific goal, one must decide on a metric by which to evaluate the quality of predictions. We consider the problem of measuring the performance of a prediction algorithm with the same data that were used to train the algorithm. Typical approaches involve bootstrapping or cross-validation. However, we demonstrate that bootstrap-based approaches often fail and standard cross-validation estimators may perform poorly. We provide a general study of cross-validation-based estimators that highlights the source of this poor performance, and propose an alternative framework for estimation using techniques from the efficiency theory literature. We provide a theorem establishing the weak convergence of our estimators. The general theorem is applied in detail to two specific examples and we discuss possible extensions to other parameters of interest. For the two explicit examples that we consider, our estimators demonstrate remarkable finite-sample improvements over standard approaches. Supplementary materials for this article are available online.