Minimum Message Length shrinkage estimation

Abstract

This note considers estimation of the mean of a multivariate Gaussian distribution with known variance within the Minimum Message Length (MML) framework. Interestingly, the resulting MML estimator exactly coincides with the positive-part James–Stein estimator under the choice of an uninformative prior. A new approach for estimating parameters and hyperparameters in general hierarchical Bayes models is also presented.

Introduction

This work considers the problem of estimating the mean of a multivariate Gaussian distribution with a known variance given a single data sample. Define $X$ as a random variable distributed according to a multivariate Gaussian density $X\sim N_k(\mu ,\Sigma )$ with an unknown mean $\mu \in {\mathbb{R}}^k$ and a known variance $\Sigma =I_k$. The accuracy, or risk, of an estimator $\hat{\mu }\left(x\right)$ of $\mu$ is defined as (Wald, 1971)

$$R(\mu ,\hat{\mu }\left(x\right))=E_x\left[L(\mu ,\hat{\mu }\left(x\right))\right]$$

where $L(\cdot )\ge 0$ is the squared error loss function:

$$L(\mu ,\hat{\mu }\left(x\right))={(\hat{\mu }\left(x\right)-\mu )}^{\prime }(\hat{\mu }\left(x\right)-\mu )\text{.}$$

The task is to find an estimator $\hat{\mu }\left(x\right)$ which minimises the risk for all values of $\mu$. Specifically, this work examines the problem of inferring the mean $\mu$ from a single observation $x\in {\mathbb{R}}^k$ of the random variable $X$.

It is well known that the Uniformly Minimum Variance Unbiased (UMVU) estimator of $\mu$ is the least squares estimate (Lehmann and Casella, 2003) given by

$${\hat{\mu }}_{\mathrm{LS}}\left(x\right)=x\text{.}$$

This estimator is minimax under the squared error loss function and is equivalent to the maximum likelihood estimator. Stein (1956) has demonstrated that, remarkably, for $k\ge 3$, the least squares estimator is not admissible and is in fact dominated by a large class of minimax estimators. The most well known of these dominating estimators is the positive-part James–Stein estimator (James and Stein, 1961):

$${\hat{\mu }}_{\mathrm{JS}}\left(x\right)={(1-\frac{k-2}{x^{\prime }x})}_{+}x$$

where ${(\cdot )}_{+}=max(0,\cdot )$. Estimators in the James–Stein class tend to shrink towards some origin (in this case zero) and hence are usually referred to as shrinkage estimators. Shrinkage estimators dominate the least squares estimator by trading some increase in bias for a larger decrease in variance. A common method for deriving the James–Stein estimator is through the empirical Bayes method (Robbins, 1964), in which the mean $\mu$ is assumed to be distributed as per $N_k(0_k,c\cdot I_k)$ and the hyperparameter $c>0$ is estimated from the data.

This work examines the James–Stein problem within the Minimum Message Length framework (see Section 2). Specifically, we derive MML estimators of $\mu$ and $c$ which exactly coincide with the positive-part James–Stein estimator under the choice of an uninformative prior over $c$ (see Section 3). A systematic approach to finding MML estimators for the parameters and hyperparameters in general hierarchical Bayes models is then developed (see Section 4). As a corollary, the new method of hyperparameter estimation appears to provide an information theoretic basis for hierarchical Bayes estimation. Some examples with multiple hyperparameters are discussed in Section 5. Concluding remarks are given in Section 6.