Occam factors and model independent Bayesian learning of continuous distributions

• Type of Material: Article
• Authors: Ilya Nemenman, Emory University; William Bialek, NEC Research Institute
• Language: English
• Date: 2002-02
• Publisher: American Physical Society
• Publication Version: Final Published Version
• Copyright Statement: © 2002 The American Physical Society
• Final Published Version (URL): http://link.aps.org/doi/10.1103/PhysRevE.65.026137
• Title of Journal or Parent Work: Physical Review E
• ISSN: 1539-3755
• Volume: 65
• Start Page: 026137
• End Page: 026137
• Grant/Funding Information: Work at Princeton was supported in part by funds from NEC.
• Research Categories: Biophysics, General

Abstract

Learning of a smooth but nonparametric probability density can be regularized using methods of quantum field theory. We implement a field theoretic prior numerically, test its efficacy, and show that the data and the phase space factors arising from the integration over the model space determine the free parameter of the theory ~‘‘smoothness scale’’! self-consistently. This persists even for distributions that are atypical in the prior and is a step towards a model independent theory for learning continuous distributions. Finally, we point out that a wrong parametrization of a model family may sometimes be advantageous for small data sets.