Sooraj Bhat
Richard Vuduc
Abstract
There has been great interest in creating probabilistic programming languages to simplify the coding of statistical tasks; however, there still does not exist a formal language that simultaneously provides (1) continuous probability distributions, (2) the ability to naturally express custom probabilistic models, and (3) probability density functions (PDFs). This collection of features is necessary for mechanizing fundamental statistical techniques. We formalize the first probabilistic language that exhibits these features, and it serves as a foundational framework for extending the ideas to more general languages. Particularly novel are our type system for absolutely continuous (AC) distributions (those which permit PDFs) and our PDF calculation procedure, which calculates PDF s for a large class of AC distributions. Our formalization paves the way toward the rigorous encoding of powerful statistical reformulations.
Supplemental Material
- A. Agarwal, S. Bhat, A. Gray, and I. E. Grossmann. Automating Mathematical Program Transformations. In Practical Aspects of Declarative Languages, 2010. Google Scholar
Digital Library
- P. Audebaud and C. Paulin-Mohring. Proofs of Randomized Algorithms in Coq. In Mathematics of Program Construction, pages 49--68. Springer, 2006. Google ScholarDigital Library
- C. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. Google ScholarDigital Library
- J. Borgstram, A. D. Gordon, M. Greenberg, J. Margetson, and J. V. Gael. Measure Transformer Semantics for Bayesian Machine Learning. In European Symposium on Programming, pages 77--96, 2011. Google ScholarDigital Library
- H. Daumé III. HBC: Hierarchical Bayes Compiler, 2007. URL http://hal3.name/HBC.Google Scholar
- L. Devroye. Non-Uniform Random Variate Generation, 1986.Google Scholar
- M. Erwig and S. Kollmansberger. Functional Pearls: Probabilistic Functional Programming in Haskell. Journal of Functional Programming, 16 (01): 21--34, 2005. Google ScholarDigital Library
- M. Giry. A Categorical Approach to Probability Theory. Categorical Aspects of Topology and Analysis, 915: 68--85, 1981.Google ScholarCross Ref
- N. Goodman, V. Mansinghka, D. Roy, K. Bonawitz, and J. Tenenbaum. Church: A Language for Generative Models. In Uncertainty in Artificial Intelligence, 2008.Google Scholar
- A. G. Gray, B. Fischer, J. Schumann, and W. Buntine. Automatic Derivation of Statistical Algorithms: The EM Family and Beyond. In Advances in Neural Information Processing Systems, 2003.Google Scholar
- M. Hoyrup, C. Rojas, and K. Weihrauch. The Radon-Nikodym operator is not computable. In Computability & Complexity in Analysis, 2011.Google Scholar
- K. Kersting and L. De Raedt. Bayesian Logic Programming: Theory and Tool. In Introduction to Statistical Relational Learning. 2007.Google Scholar
- O. Kiselyov and C. Shan. Embedded Probabilistic Programming. In Working Conference on Domain Specific Languages. Springer, 2009. Google ScholarDigital Library
- D. Kozen. Semantics of Probabilistic Programs. Journal of Computer and System Sciences, 22 (3): 328--350, 1981.Google ScholarCross Ref
- T. Mhamdi, O. Hasan, and S. Tahar. On the Formalization of the Lebesgue Integration Theory in HOL. Interactive Theorem Proving, pages 387--402, 2010. Google ScholarDigital Library
- B. Milch, B. Marthi, S. Russell, D. Sontag, D. Ong, and A. Kolobov. BLOG: Probabilistic Models with Unknown Objects. In International Joint Conference on Artificial Intelligence, volume 19, 2005. Google ScholarDigital Library
- O. Nielsen. An Introduction to Integration and Measure Theory. Wiley-Interscience, 1997.Google Scholar
- S. Park, F. Pfenning, and S. Thrun. A Probabilistic Language based upon Sampling Functions. In Principles of Programming Languages, pages 171--182. ACM New York, NY, USA, 2005. Google ScholarDigital Library
- A. Pfeffer. IBAL: A Probabilistic Rational Programming Language. In International Joint Conference on Artificial Intelligence, 2001. Google ScholarDigital Library
- N. Ramsey and A. Pfeffer. Stochastic Lambda Calculus and Monads of Probability Distributions. volume 37, pages 154--165. ACM, 2002. Google ScholarDigital Library
- M. Richardson and P. Domingos. Markov Logic Networks. Machine Learning, 62 (1): 107--136, 2006. Google ScholarDigital Library
- T. Sato and Y. Kameya. PRISM: A Symbolic-Statistical Modeling Language. In International Joint Conference on Artificial Intelligence, pages 1330--1339, 1997. Google ScholarDigital Library
- D. Scott. Parametric Statistical Modeling by Minimum Integrated Square Error. Technometrics, 43 (3): 274--285, 2001.Google ScholarCross Ref
- B. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall/CRC, 1986.Google ScholarCross Ref
- R. Solovay. A Model of Set-Theory in Which Every Set of Reals is Lebesgue Measurable. Annals of Mathematics, pages 1--56, 1970.Google ScholarCross Ref
- L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004. Google ScholarDigital Library
Index Terms
- A type theory for probability density functions
Recommendations
Deriving a probability density calculator (functional pearl)
ICFP 2016: Proceedings of the 21st ACM SIGPLAN International Conference on Functional ProgrammingGiven an expression that denotes a probability distribution, often we want a corresponding density function, to use in probabilistic inference. Fortunately, the task of finding a density has been automated. It turns out that we can derive a ...
A type theory for probability density functions
POPL '12: Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languagesThere has been great interest in creating probabilistic programming languages to simplify the coding of statistical tasks; however, there still does not exist a formal language that simultaneously provides (1) continuous probability distributions, (2) ...
Deriving a probability density calculator (functional pearl)
ICFP '16Given an expression that denotes a probability distribution, often we want a corresponding density function, to use in probabilistic inference. Fortunately, the task of finding a density has been automated. It turns out that we can derive a ...
Comments