Post-hoc power analysis: a conceptually valid approach for power based on observed study data

• Type of Material: Article
• Authors: Natalie E Quach, Human Longevity, Inc.; Kun Yang, Human Longevity, Inc.; Ruohui Chen, Human Longevity, Inc.; Justin Tu, Emory University; Manfei Xu, Shanghai Jiao Tong University School of Medicine; Xin M Zhang, Human Longevity, Inc.
• Language: English
• Date: 2022-09-13
• Publisher: BMJ.
• Publication Version: Final Published Version
• Copyright Statement: © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
• License: Creative Commons Attribution-NonCommercial 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.1136/gpsych-2022-100764
• Title of Journal or Parent Work: General Psychiatry
• Volume: 35
• Issue: 4
• Start Page: e100764
• End Page: e100764
• Grant/Funding Information: The project described was partially supported by the National Institutes of Health (grant UL1TR001442) of CTSA funding.
• Keywords: Observational Studies as Topic; Statistics as Topic; Power, Psychological; Biostatistics
• Research Categories: Health Sciences, Mental Health

Abstract

Power analysis is a key component of planning prospective studies such as clinical trials. However, some journals in biomedical and psychosocial sciences request power analysis for data already collected and analysed before accepting manuscripts for publication. Many have raised concerns about the conceptual basis for such post-hoc power analyses. More recently, Zhang et al showed by using simulation studies that such power analyses do not indicate true power for detecting statistical significance since post-hoc power estimates vary in the range of practical interests and can be very different from the true power. On the other hand, journals' request for information about the reliability of statistical findings in a manuscript due to small sample sizes is justified since the sample size plays an important role in the reproducibility of statistical findings. The problem is the wording of the journals' request, as the current power analysis paradigm is not designed to address journals' concerns about the reliability of the statistical findings. In this paper, we propose an alternate formulation of power analysis to provide a conceptually valid approach to the journals' wrongly worded but practically significant concern.