Abstract
This paper derives two new information theoretic linear regression criteria based on the minimum message length principle. Both criteria are invariant to full rank affine transformations of the design matrix and yield estimates that are minimax with respect to squared error loss. The new criteria are compared against state of the art information theoretic model selection criteria on both real and synthetic data and show good performance in all cases.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Wallace, C.S.: Statistical and Inductive Inference by Minimum Message Length, 1st edn. Information Science and Statistics. Springer, Heidelberg (2005)
Fitzgibbon, L.J., Dowe, D.L., Allison, L.: Univariate polynomial inference by Monte Carlo message length approximation. In: 19th International Conference on Machine Learning (ICML 2002), Sydney, Australia, pp. 147–154 (2002)
Viswanathan, M., Wallace, C.S.: A note on the comparison of polynomial selection methods. In: Uncertainty 1999: The Seventh International Workshop on Artificial Intelligence and Statistics, Fort Lauderdale, Florida, pp. 169–177 (1999)
Rissanen, J.: Information and Complexity in Statistical Modeling, 1st edn. Information Science and Statistics. Springer, Heidelberg (2007)
Wallace, C.S., Boulton, D.M.: An information measure for classification. Computer Journal 11(2), 185–194 (1968)
Wallace, C.S., Freeman, P.R.: Estimation and inference by compact coding. Journal of the Royal Statistical Society (Series B) 49(3), 240–252 (1987)
Makalic, E., Schmidt, D.F.: Minimum message length shrinkage estimation. Statistics & Probability Letters 79(9), 1155–1161 (2009)
Akaike, H.: A new look at the statistical model identification. IEEE Transactions on Automatic Control 19(6), 716–723 (1974)
Schwarz, G.: Estimating the dimension of a model. The Annals of Statistics 6(2), 461–464 (1978)
Zellner, A.: Applications of Bayesian analysis in econometrics. The Statistician 32(1–2), 23–34 (1983)
Sclove, S.L.: Improved estimators for coefficients in linear regression. Journal of the American Statistical Association 63(322), 596–606 (1968)
Roos, T., Myllymäki, P., Rissanen, J.: MDL denoising revisited. IEEE Transactions on Signal Processing 57(9), 3347–3360 (2009)
Hansen, M.H., Yu, B.: Model selection and the principle of minimum description length. Journal of the American Statistical Association 96(454), 746–774 (2001)
Seghouane, A.K., Bekara, M.: A small sample model selection criterion based on Kullback’s symmetric divergence. IEEE Trans. Sig. Proc. 52(12), 3314–3323 (2004)
Asuncion, A., Newman, D.: UCI machine learning repository (2007)
Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. The Annals of Statistics 32(2), 407–451 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Schmidt, D.F., Makalic, E. (2009). MML Invariant Linear Regression. In: Nicholson, A., Li, X. (eds) AI 2009: Advances in Artificial Intelligence. AI 2009. Lecture Notes in Computer Science(), vol 5866. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10439-8_32
Download citation
DOI: https://doi.org/10.1007/978-3-642-10439-8_32
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-10438-1
Online ISBN: 978-3-642-10439-8
eBook Packages: Computer ScienceComputer Science (R0)