Robust Lasso Regression with Student-t Residuals

Abstract

The lasso, introduced by Robert Tibshirani in 1996, has become one of the most popular techniques for estimating Gaussian linear regression models. An important reason for this popularity is that the lasso can simultaneously estimate all regression parameters as well as select important variables, yielding accurate regression models that are highly interpretable. This paper derives an efficient procedure for fitting robust linear regression models with the lasso in the case where the residuals are distributed according to a Student-t distribution. In contrast to Gaussian lasso regression, the proposed Student-t lasso regression procedure can be applied to data sets which contain large outlying observations. We demonstrate the utility of our Student-t lasso regression by analysing the Boston housing data set.

Appendix A

To find an appropriate maximum value $\tau _\mathrm{max}$ of $\tau$ for producing a regularisation path we use the following heuristic procedure: let $\hat{\varvec{\beta }}_\mathrm{ML}$ and $\hat{\sigma }^2_\mathrm{ML}$ denote the maximum likelihood estimates for $\varvec{\beta }$ and $\sigma ^2$. The negative log-prior probability of the maximum likelihood estimates, under the Laplace prior (3), is given by

$$ p \log (\tau ) + \left( \frac{\sqrt{2}}{\tau \hat{\sigma }_\mathrm{ML}} \right) || \hat{\varvec{\beta }}_\mathrm{ML} ||_1 + \mathrm{const}, $$

where $\mathrm{const}$ denotes terms that do not depend on either $\tau$ or $\hat{\varvec{\beta }}_\mathrm{ML}$. The value of $\tau$ that maximises the prior probability for $\hat{\varvec{\beta }}_\mathrm{ML}$ is given by

$$ \tilde{\tau } = \frac{\sqrt{2} \, ||\hat{\varvec{\beta }}_\mathrm{ML}||_1}{p \hat{\sigma }_\mathrm{ML}}. $$

We then choose $\tau _\mathrm{max} = c \tilde{\tau }$, where $c>1$ is a constant that controls the distance of the maximum likelihood estimates to the final point on the regularisation path.