François Caron
ABSTRACT
One of the most common problems in machine learning and statistics consists of estimating the mean response Xβ from a vector of observations y assuming y = Xβ + ε where X is known, β is a vector of parameters of interest and ε a vector of stochastic errors. We are particularly interested here in the case where the dimension K of β is much higher than the dimension of y. We propose some flexible Bayesian models which can yield sparse estimates of β. We show that as K → ∞ these models are closely related to a class of Lévy processes. Simulations demonstrate that our models outperform significantly a range of popular alternatives.
- Applebaum, D. (2004). Lévy processes and stochastic calculus. Cambridge University Press.Google Scholar
- Barndorff-Nielsen, O. (1997). Normal inverse Gaussian distributions and stochastic volatility modelling. Scandinavian Journal of Statistics, 24, 1--13.Google ScholarCross Ref
- Chen, S., Donoho, D., & Saunders, M. (2001). Atomic decomposition by basis pursuit. SIAM review, 43, 129--159. Google ScholarDigital Library
- Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96, 1348--1360.Google ScholarCross Ref
- Figueiredo, M. (2003). Adaptive sparseness for supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25, 1150--1159. Google ScholarDigital Library
- Ghahramani, Z., Griffiths, T., & Sollich, P. (2006). Bayesian nonparametric latent feature models. Proceedings Valencia/ISBA World meeting on Bayesian statistics.Google Scholar
- Girolami, M. (2001). A variational method for learning sparse and overcomplete representations. Neural computation, 13, 2517--2532. Google ScholarDigital Library
- Griffin, J., & Brown, P. (2007). Bayesian adaptive lasso with non-convex penalization (Technical Report). Dept of Statistics, University of Warwick.Google Scholar
- Lewicki, M. S., & Sejnowski, T. (2000). Learning over-complete representations. Neural computation, 12, 337--365. Google ScholarDigital Library
- Madan, D., & Seneta, E. (1990). The variance-gamma model for share market returns. Journal of Business, 63, 511--524.Google ScholarCross Ref
- Snoussi, H., & Idier, J. (2006). Bayesian blind separation of generalized hyperbolic processes in noisy and underdeterminate mixtures. IEEE Transactions on Signal Processing, 54, 3257--3269. Google ScholarDigital Library
- Teh, Y., Gorur, D., & Ghahramani, Z. (2007). Stick-breaking construction for the Indian buffet process. International Conference on Artificial Intelligence and Statistics.Google Scholar
- Thibaux, R., & Jordan, M. (2007). Hierarchical beta processes and the Indian buffet process. International Conference on Artificial Intelligence and Statistics.Google Scholar
- Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society B, 58, 267--288.Google ScholarCross Ref
- Tipping, M. (2001). Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 211--244, 211--244. Google ScholarDigital Library
- Titsias, M. (2007). The infinite gamma-Poisson feature model. International Conference on Neural Information Processing Systems.Google Scholar
- Tsilevich, N., Vershik, A., & Yor, M. (2000). Distinguished properties of the gamma process, and related topics (Technical Report). Laboratoire de Probabilités et Modèles aléatoires, Paris.Google Scholar
- West, M. (2003). Bayesian factor regression models in the "Large p, Small n" paradigm. In J. Bernardo, M. Bayarri, J. Berger, A. Dawid, D. Heckerman, A. Smith and M. West (Eds.), Bayesian statistics 7, 723--732. Oxford University Press.Google Scholar
Index Terms
- Sparse Bayesian nonparametric regression
Recommendations
Testing independence in nonparametric regression
We propose a new test for independence of error and covariate in a nonparametric regression model. The test statistic is based on a kernel estimator for the L"2-distance between the conditional distribution and the unconditional distribution of the ...
Analysis of Variance in Nonparametric Regression Models
In a nonparametric regression model with a multivariate explanatory variable we consider the problem of testing the hypothesis that specific interactions in a canonical decomposition of the model vanish. A simple consistent test is developed which is ...
Bayesian nonparametric covariance regression
Capturing predictor-dependent correlations amongst the elements of a multivariate response vector is fundamental to numerous applied domains, including neuroscience, epidemiology, and finance. Although there is a rich literature on methods for allowing ...
Comments