The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Comparison of Laplace and Double Pareto Penalty: LASSO and Elastic Net

라플라스와 이중 파레토 벌점의 비교: LASSO와 Elastic Net

  • Kyung, Minjung (Department of Statistics, Duksung Women's University)
  • 경민정 (덕성여자대학교 정보통계학과)
  • Received : 2014.09.22
  • Accepted : 2014.10.21
  • Published : 2014.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Lasso (Tibshirani, 1996) and Elastic Net (Zou and Hastie, 2005) have been widely used in various fields for simultaneous variable selection and coefficient estimation. Bayesian methods using a conditional Laplace and a double Pareto prior specification have been discussed in the form of hierarchical specification. Full conditional posterior distributions with each priors have been derived. We compare the performance of Bayesian lassos with Laplace prior and the performance with double Pareto prior using simulations. We also apply the proposed Bayesian hierarchical models to real data sets to predict the collapse of governments in Asia.

연속적인 변수 선택과 계수 추정을 동시에 활용할 수 있다는 특성 때문에 LASSO (Tibshirani, 1996)와 Elastic Net (Zou와 Hastie, 2005)은 다양한 분야에서 활발하게 사용되고 있다. 조건부 라플라스와 이중 파레토 사전분포를 적용한 공액계층모형을 표현하였고, 각각의 사전분포에 대한 완전 조건 사후분포를 도출하였다. 제안된 사전분포를 적용한 벌점회귀모형을 비교하기 위한 모의 실험을 진행하였고, 예측정확도를 판단하기 위해 아시아 국가 실패(the collapse of governments in Asia)의 실제 데이터에 제안한 모형을 적용하였다.

Keywords

References

  1. Andrews, D. F. and Mallows, C. L. (1974). Scale mixtures of normal distributions, Journal of the Royal Statistical Society, Series B, 36, 99-102.
  2. Armagan, A., Dunson, D. B. and Lee, J. (2013). Generalized double Pareto shrinkage, Statistica Sinica, 23, 119-143.
  3. Bae, K. and Mallick, B. K. (2004). Gene selection using a two-level hierarchical Bayesian model, Bioinfor- matics, 20, 3423-3430. https://doi.org/10.1093/bioinformatics/bth419
  4. Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression, The Annals of Statistics, 32, 407-451. https://doi.org/10.1214/009053604000000067
  5. Esty, D. C., Goldstone, J. A., Gurr, T. R., Harff, B., Levy, M., Dabelko, G. D., Surko, P. T. and Unger, A. N. (1999). State Failure Task Force Report: Phase II Findings Environmental Change & Security Project Report 5, Summer.
  6. Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, 96, 1348-1360. https://doi.org/10.1198/016214501753382273
  7. Figueiredo, M. A. T. (2003). Adaptive sparseness for supervised learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25 1150-1159. https://doi.org/10.1109/TPAMI.2003.1227989
  8. Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics regression tools, Technometrics, 35, 109-135. https://doi.org/10.1080/00401706.1993.10485033
  9. Hobert, J. P. and Geyer, C. J. (1998). Geometric ergodicity of Gibbs and Block Gibbs samplers for a hierarchical random effects model, Journal of Multivariate Analysis, 67 414-430. https://doi.org/10.1006/jmva.1998.1778
  10. Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Applications to nonorthogonal problems, Technometrics, 12, 55-68. https://doi.org/10.1080/00401706.1970.10488634
  11. Kim, Y., Kim, J. and Kim, Y. (2006). Blockwise sparse regression, Statistica Sinica, 16, 375-390.
  12. Knight, K. and Fu, W. (2000). Asymptotics for lasso-type estimators, The Annals of Statistics, 28, 1356-1378. https://doi.org/10.1214/aos/1015957397
  13. Kyung, M., Gill, J., Ghosh, M. and Casella, G. (2010). Penalized regression, standard errors, and Bayesian lassos, Bayesian Analysis, 5, 369-412. https://doi.org/10.1214/10-BA607
  14. Liu, J. S. , Wong, W. H. and Kong, A. (1994). Covariance structure of the Gibbs sampler with applications to the comparison of estimators and augmentation schemes, Biometrika, 81, 27-40. https://doi.org/10.1093/biomet/81.1.27
  15. Osborne, M. R., Presnell, B. and Turlach, B. A. (2000). A new approach to variable selection in least squares problems, IMA Journal of Numerical Analysis, 20, 389-404. https://doi.org/10.1093/imanum/20.3.389
  16. Park, T. and Casella, G. (2008). The Bayesian lasso, Journal of the American Statistical Association, 103, 681-686. https://doi.org/10.1198/016214508000000337
  17. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society, Series B, 58, 267-288.
  18. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005). Sparsity and smoothness via the fused lasso, Journal of the Royal Statistical Society, Series B, 67, 91-108. https://doi.org/10.1111/j.1467-9868.2005.00490.x
  19. Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables, Journal of the Royal Statistical Society, Series B, 68, 49-67. https://doi.org/10.1111/j.1467-9868.2005.00532.x
  20. Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net, Journal of the Royal Statistical Society, Series B, 67, 301-320. https://doi.org/10.1111/j.1467-9868.2005.00503.x