Sufficient Statistics, Exponential Families, and Estimation

Abstract

A sufficient statistic is a function of the observed data X containing all the information that X holds about the model. A complete sufficient statistic is one that reduces the data the most, without losing any information. More importantly, according to Rao–Blackwell-, Lehmann–Scheffé-theorems, statistical inference procedures must be based on such statistics for purposes of efficiency or optimality.

Appendix for Project: The Nonparametric Percentile Bootstrap of Efron

Let $\hat{\sigma }_{n}$ be the standard error of $\hat{\theta }_{n}$ (That is, $\hat{\sigma }_{n}$ is an estimate of the standard deviation of $\hat{\theta }_{n}$). An asymptotic confidence interval of confidence level 1 −α for θ would follow from the relation $P(z_{\alpha /2}\hat{\sigma }_{n} \leq \hat{\theta }_{n}-\theta \leq z_{1-\alpha /2}\hat{\sigma }_{n}) \approx 1-\alpha$, namely, it is the interval $[\hat{\theta }_{n} - z_{1-\alpha /2}\hat{\sigma }_{n},\hat{\theta }_{n} - z_{\alpha /2}\hat{\sigma }_{n}] = [\hat{\theta }_{n} - z_{\alpha /2}\hat{\sigma }_{n},\hat{\theta }_{n} + z_{1-\alpha /2}\hat{\sigma }_{n}] = [l,u]$, say. Now the bootstrap version $\hat{\theta }_{n}^{{\ast}}$ of $\hat{\theta }_{n}$ is, under the empirical $\mathbf{P}^{{\ast}} =\widehat{ P}_{n}$, asymptotically Normal $N(\hat{\theta }_{n},\hat{\sigma }_{n}^{2})$, so that the α∕2-th and (1 −α∕2)-th quantiles of $\hat{\theta }_{n}^{{\ast}}$, q _α∕2 ^∗ and q _1−α∕2 ^∗ say, are asymptotically equal to $\hat{\theta }_{n} + z_{\alpha /2}\hat{\sigma }_{n} = l$ and $\hat{\theta }_{n} + z_{1-\alpha /2}\hat{\sigma }_{n} = u$, respectively.

Hence the percentile bootstrap based confidence interval for θ is given by

$$\displaystyle{ \left [q_{\alpha /2}^{{\ast}},q_{ 1-\alpha /2}^{{\ast}}\right ]. }$$

(4.62)

Note that the construction of this interval only involves resampling from the data repeatedly to construct bootstrap versions $\hat{\theta }_{n}^{{\ast}}$ of $\hat{\theta }_{n}$; it does not involve the computation of the standard error $\hat{\theta }_{n}$.

Although ( 4.62) does not involve computing the standard error $\hat{\sigma }_{n}$, the latter is an important object in statistical analysis. It follows from the above that the variance $\hat{\sigma }_{n}^{{\ast}2}$ of the $\hat{\theta }_{n}^{{\ast}}$ values from the repeated resamplings provide an estimate of $\hat{\sigma }_{n}^{2}$ [A rough estimate of $\hat{\sigma }_{n}$ is also provided by $([q_{1-\alpha /2}^{{\ast}}- q_{\alpha /2}^{{\ast}}]/2z_{1-\alpha /2})^{\frac{1} {2} }]$.

When the standard error $\hat{\sigma }_{n}$ of $\hat{\theta }_{n}$ is known in closed form, one may use the studentized or pivoted statistic $T_{n} = (\hat{\theta }_{n}-\theta )/\hat{\sigma }_{n}$ which is asymptotically standard Normal N(0, 1). The usual CLT-based symmetric confidence interval for θ is given by

$$\displaystyle{ \left [\hat{\theta }_{n} + z_{\alpha /2}\hat{\sigma }_{n},\hat{\theta }_{n} + z_{1-\alpha /2}\hat{\sigma }_{n}\right ] = \left [\hat{\theta }_{n} - z_{1-\alpha /2}\hat{\sigma }_{n},\hat{\theta }_{n} - z_{\alpha /2}\hat{\sigma }_{n}\right ], }$$

(4.63)

using P( | T _n  | ≤ z _1−α∕2) = 1 −α. The corresponding pivotal bootstrap confidence interval is based on the resampled values of $T_{n}^{{\ast}} = (\hat{\theta }_{n}^{{\ast}}-\hat{\theta }_{n})/\hat{\sigma }_{n}^{{\ast}}$, where $\hat{\sigma }_{n}^{{\ast}}$ is the bootstrap estimate of the standard error as described in the preceding paragraph. Let c _α∕2 ^∗ be such that P ^∗( | T _n ^∗ | ≤ c _α∕2 ^∗) = 1 −α. The bootstrap pivotal confidence interval for θ is then

$$\displaystyle{ \left [\hat{\theta }_{n} - c_{\alpha /2}^{{\ast}}\hat{\sigma }_{ n}^{{\ast}},\hat{\theta }_{ n} + c_{\alpha /2}^{{\ast}}\hat{\sigma }_{ n}^{{\ast}}\right ]. }$$

(4.64)

Suppose $\hat{\theta }_{n}$ is based on i.i.d. observations $X_{1},\ldots,X_{n}$, whose common distribution has a density (or a nonzero density component), and that it is a smooth function of sample means of a finite number of characteristics of X, or has a stochastic expansion (Taylor expansion) in terms of these sample means (such as the MLE in regular cases). It may then be shown that the coverage error of the CLT-based interval ( 4.63) is O(n ⁻¹), while that based on ( 4.64) is O(n ^−3∕2), a major advantage of the bootstrap procedure. The coverage error of the percentile interval ( 4.62) is O(n ^−1∕2), irrespective of whether the distribution of X is continuous or discrete.

Definition 4.6.

The coverage error of a confidence interval for a parameter θ is the (absolute) difference between the actual probability that the true parameter value belongs to the interval and the target level 1 −α.