- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- BIRS Workshop Lecture Videos /
- Annealing Between Distributions by Averaging Moments
Open Collections
BIRS Workshop Lecture Videos
BIRS Workshop Lecture Videos
Annealing Between Distributions by Averaging Moments Salakhutdinov, Russ
Description
Many powerful Monte Carlo techniques for estimating partition functions, such as annealed importance sampling (AIS), are based on sampling from a sequence of intermediate distributions which interpolate between a tractable initial distribution and the intractable target distribution. The near-universal practice is to use geometric averages of the initial and target distributions, but alternative paths can perform substantially better. We present a novel sequence of intermediate distributions for exponential families defined by averaging the moments of the initial and target distributions. We analyze the asymptotic performance of both the geometric and moment averages paths and derive an asymptotically optimal piecewise linear schedule. AIS with moment averaging performs well empirically at estimating partition functions of restricted Boltzmann machines (RBMs), which form the building blocks of many deep learning models, including Deep Belief Networks and Deep Boltzmann Machines. Joint work with Roger Grosse and Chris Maddison. References: Annealing between Distributions by Averaging Moments. Roger Grosse, Chris Maddison, and Ruslan Salakhutdinov. In Neural Information Processing Systems (NIPS 27) www.cs.toronto.edu/~rsalakhu/papers/nips2013_moment.pdf
Item Metadata
Title |
Annealing Between Distributions by Averaging Moments
|
Creator | |
Publisher |
Banff International Research Station for Mathematical Innovation and Discovery
|
Date Issued |
2014-02-13
|
Description |
Many powerful Monte Carlo techniques for estimating partition functions, such as annealed importance sampling (AIS), are based on sampling from a sequence of intermediate distributions which interpolate between a tractable initial distribution and the intractable target distribution. The near-universal practice is to use geometric averages of the initial and target distributions, but alternative paths can perform substantially better. We present a novel sequence of intermediate distributions for exponential families defined by averaging the moments of the initial and target distributions. We analyze the asymptotic performance of both the geometric and moment averages paths and derive an asymptotically optimal piecewise linear schedule. AIS with moment averaging performs well empirically at estimating partition functions of restricted Boltzmann machines (RBMs), which form the building blocks of many deep learning models, including Deep Belief Networks and Deep Boltzmann Machines. Joint work with Roger Grosse and Chris Maddison.
References:
Annealing between Distributions by Averaging Moments. Roger Grosse, Chris Maddison, and Ruslan Salakhutdinov. In Neural Information Processing Systems (NIPS 27) www.cs.toronto.edu/~rsalakhu/papers/nips2013_moment.pdf
|
Extent |
40 minutes
|
Subject | |
Type | |
File Format |
video/mp4
|
Language |
eng
|
Notes |
Author affiliation: University of Toronto
|
Series | |
Date Available |
2014-08-07
|
Provider |
Vancouver : University of British Columbia Library
|
Rights |
Attribution-NonCommercial-NoDerivs 2.5 Canada
|
DOI |
10.14288/1.0043898
|
URI | |
Affiliation | |
Peer Review Status |
Unreviewed
|
Scholarly Level |
Faculty
|
Rights URI | |
Aggregated Source Repository |
DSpace
|
Item Media
Item Citations and Data
Rights
Attribution-NonCommercial-NoDerivs 2.5 Canada