Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results

• Type of Material: Article
• Authors: Forike K. Martens, VU University Medical Center; Elisa C. M. Tonk, VU University Medical Center; A Cecile Janssens, Emory University
• Language: English
• Date: 2018-06-12
• Publisher: Springer Nature [academic journals on nature.com]: Hybrid Journals - choice of CC licence
• Publication Version: Author Accepted Manuscript (After Peer Review)
• Copyright Statement: © 2018, Springer Nature
• Final Published Version (URL): http://dx.doi.org/10.1038/s41436-018-0058-9
• Title of Journal or Parent Work: Genetics in Medicine
• ISSN: 1098-3600
• Volume: 21
• Issue: 2
• Start Page: 391
• End Page: 397
• Grant/Funding Information: This work was supported by a consolidator grant from the European Research Council (GENOMICMEDICINE).
• Keywords: polygenic; risk prediction; integrated discrimination improvement; net reclassification improvement; area under the curve
• Research Categories: Health Sciences, Public Health; Biology, Genetics

Abstract

Purpose The area under the receiver operating characteristic curve (AUC) is commonly used for evaluating the improvement of polygenic risk models and increasingly assessed together with the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We evaluated how researchers described and interpreted AUC, NRI, and IDI when simultaneously assessed. Methods We reviewed how researchers described definitions of AUC, NRI and IDI and how they computed each metric. Next, we reviewed how the increment in AUC, NRI and IDI were interpreted; and how the overall conclusion about the improvement of the risk model was reached. Results AUC, NRI and IDI were correctly defined in 63%, 70%, and 0% of the articles. All statistically significant values and almost half of the non-significant were interpreted as indicative of improvement, irrespective of the values of the metrics. Also, small, nonsignificant changes in the AUC were interpreted as indication of improvement when NRI and IDI were statistically significant. Conclusion Researchers have insufficient knowledge about how to interpret the various metrics for the assessment of the predictive performance of polygenic risk models and rely on the statistical significance for their interpretation. A better understanding is needed to achieve more meaningful interpretation of polygenic prediction studies.