A simple and useful regression model for underdispersed count data based on Bernoulli–Poisson convolution

Abstract

Count data is often modeled using Poisson regression, although this probability model naturally restricts the conditional variance to be equal to the conditional mean (equidispersion property). While overdispersion has been intensively studied, there are few alternative models in the statistical literature for analyzing count data with underdispersion. The primary goal of this paper is to introduce a novel model based on Bernoulli-Poisson convolution for modelling count data that are underdispersed relative to the Poisson distribution. We study the statistical properties of the proposed model, and we provide a useful interpretation of the parameters. We consider a regression structure for both components based on a new parameterization indexed by mean and dispersion parameters. An expectation-maximization (EM) algorithm is proposed for parameter estimation and some diagnostic measures, based on the EM algorithm, are considered. Simulation studies are conducted to evaluate its finite sample performance. Finally, we illustrate the usefulness of the new regression model by an application.

Appendix

In this section, we provided explicit expression for the terms involved in Eq. (8) when $g_1$ and $g_2$ are the logarithmic and logit functions, respectively. The first derivative of the complete log-likelihood in relation to ${{\varvec{\theta }}}$ is given by

$$\begin{aligned}&S_i({{\varvec{\theta }}})=\frac{\partial \ell _c({{\varvec{\theta }}})}{\partial {{\varvec{\theta }}}}=\left( \begin{array}{c} \displaystyle \sum _{i=1}^n {\mathbf {x}}_i^\top (a_i+b_i t_i) \\ \displaystyle \sum _{i=1}^n {\mathbf {z}}_i^\top (c_i+d_i t_i) \end{array} \right) \Rightarrow \text {E}\left( S_i({{\varvec{\theta }}})\mid {{\varvec{D}}}_{obs}; {{\varvec{\theta }}}=\widehat{{\varvec{\theta }}}\right) \\&\quad \quad =\left( \begin{array}{c} \displaystyle \sum _{i=1}^n {\mathbf {x}}_i^\top (a_i+b_i {\widetilde{t}}_i) \\ \displaystyle \sum _{i=1}^n {\mathbf {z}}_i^\top (c_i+d_i {\widetilde{t}}_i) \end{array} \right) , \end{aligned}$$

where

$$\begin{aligned} a_i&= \mu _i \left( -1+\frac{y_i}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}\right. \\&\quad \left. -\frac{1}{2}\sqrt{\frac{1-\phi _i}{\mu _i}}\left\{ -1+\frac{y_i}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}+\frac{1}{1-\sqrt{\mu _i(1-\phi _i)}}\right\} \right) ,\\ b_i&=\mu _i \left( \frac{1}{2\mu _i}-\frac{1}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}\right. \\&\quad \left. +\frac{1}{2}\sqrt{\frac{1-\phi _i}{\mu _i}}\left\{ \frac{1}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}+\frac{1}{1-\sqrt{\mu _i(1-\phi _i)}}\right\} \right) ,\\ c_i&=\frac{1}{2}\phi _i\sqrt{\mu _i(1-\phi _i)}\left( -1+\frac{y_i}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}+\frac{1}{1-\sqrt{\mu _i(1-\phi _i)}}\right) , \quad \hbox {and}\\ d_i&=-\frac{1}{2}\phi _i(1-\phi _i)\\&\quad \left( \frac{1}{(1+\phi _i)}+\sqrt{\frac{\mu _i}{1-\phi _i}}\left\{ \frac{1}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}+\frac{1}{1-\sqrt{\mu _i(1-\phi _i)}}\right\} \right) . \end{aligned}$$

On the other hand, as $\text {E}(T_i^2\mid {{\varvec{D}}}_{obs}; {{\varvec{\theta }}}=\widehat{{\varvec{\theta }}})=\text {E}(T_i\mid {{\varvec{D}}}_{obs}; {{\varvec{\theta }}}=\widehat{{\varvec{\theta }}})={\widetilde{t}}_i$, we have that

$$\begin{aligned}&\text {E}\left( S_i({{\varvec{\theta }}})S_i^\top ({{\varvec{\theta }}})\mid {{\varvec{D}}}_{obs}; {{\varvec{\theta }}}=\widehat{{\varvec{\theta }}}\right) \\&\quad =\left( \begin{array}{cc} a_i^\top a_i+{\widetilde{t}}_i(2a_ib_i^\top +b_ib_i^\top ) &{} a_i c_i^\top +{\widetilde{t}}_i (a_id_i+b_ic_i+b_id_i) \\ \cdot &{} c_i^\top c_i+{\widetilde{t}}_i(2c_id_i^\top +d_id_i^\top ) \\ \end{array} \right) . \end{aligned}$$

Finally, the expected value of the second derivative of the complete log-likelihood in relation to ${{\varvec{\theta }}}$ is given by

$$\begin{aligned} \text {E}\left( B_i({{\varvec{\theta }}})\mid {{\varvec{D}}}_{obs}, {{\varvec{\theta }}}=\widehat{{\varvec{\theta }}}\right)&=\text {E}\left( \frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial {{\varvec{\theta }}}\partial {{\varvec{\theta }}}^\top }\mid {{\varvec{D}}}_{obs}, {{\varvec{\theta }}}=\widehat{{\varvec{\theta }}}\right) \\&=\left( \begin{array}{cc} \displaystyle \sum _{i=1}^n {\mathbf {x}}_i^\top {\mathbf {x}}_iB_{11i} &{} \displaystyle \sum _{i=1}^n {\mathbf {x}}_i^\top {\mathbf {z}}_i B_{12i}\\ \displaystyle \sum _{i=1}^n {\mathbf {z}}_i^\top {\mathbf {x}}_i B_{12i} &{} \displaystyle \sum _{i=1}^n {\mathbf {z}}_i^\top {\mathbf {z}}_i B_{22i} \end{array} \right) , \end{aligned}$$

where

$$\begin{aligned} B_{11i}&=a_i+b_i {\widetilde{t}}_i+\mu _i^2 \frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial \mu _i^2}, \qquad B_{12i}=\mu _i \phi _i(1-\phi _i)\frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial \mu _i \partial \phi _i},\\ B_{22i}&=(1-2\phi _i)(c_i+d_i {\widetilde{t}}_i)+\phi _i^2(1-\phi _i)^2\frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial \phi _i^2},\\ \frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial \mu _i^2}&=-\frac{{\widetilde{t}}_i}{2\mu _i^2}-\frac{(y_i-{\widetilde{t}}_i) \left( 1-\frac{1}{2}\sqrt{\frac{1-\phi _i}{\mu _i}}\right) }{ (\mu _i-\sqrt{\mu _i(1-\phi _i)})^2}\\&\quad +\frac{1}{2}\sqrt{\frac{1-\phi _i}{\mu _i}}\Bigg (\frac{1}{2\mu _i}\bigg \{-1+\frac{y_i-{\widetilde{t}}_i}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}\\&~~~~~+\frac{1-{\widetilde{t}}_i}{1-\sqrt{\mu _i(1-\phi _i)}}\bigg \} +\frac{(y_i-{\widetilde{t}}_i)\left( 1-\frac{1}{2}\sqrt{\frac{1-\phi _i}{\mu _i}}\right) }{(\mu _i-\sqrt{\mu _i(1-\phi _i)})^2}-\frac{(1-{\widetilde{t}}_i)\sqrt{\frac{1-\phi _i}{\mu _i}}}{(1-\sqrt{\mu _i(1-\phi _i)})^2}\Bigg ),\\ \frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial \phi _i^2}&=- \frac{{\widetilde{t}}_i}{2(1-\phi _i)^2}\\&\quad +\frac{1}{4}\sqrt{\frac{\mu _i}{1-\phi _i}}\Bigg (\frac{1}{(1-\phi _i)}\left\{ -1+\frac{y_i-{\widetilde{t}}_i}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}+\frac{1-{\widetilde{t}}_i}{1-\sqrt{\mu _i(1-\phi _i)}}\right\} \\&~~~~~-\frac{(y_i-{\widetilde{t}}_i)}{(\mu _i-\sqrt{\mu _i(1-\phi _i)})^2}-\frac{(1-{\widetilde{t}}_i)}{(1-\sqrt{\mu _i(1-\phi _i)})^2}\Bigg ), \quad \hbox {and}\\ \frac{\partial ^2 \ell _c({{\varvec{\theta }}})}{\partial \mu _i \partial \phi _i}&=\frac{1}{4\sqrt{\mu _i(1-\phi _i)}}\left\{ -1+\frac{y_i-{\widetilde{t}}_i}{\mu _i-\sqrt{\mu _i(1-\phi _i)}}+\frac{1-{\widetilde{t}}_i}{1-\sqrt{\mu _i(1-\phi _i)}}\right\} \\&~~~~~-\frac{1}{4}\frac{\mu _i}{(1-\phi _i)}\left\{ \frac{(y_i-{\widetilde{t}}_i)}{(\mu _i-\sqrt{\mu _i(1-\phi _i)})^2}+\frac{(1-{\widetilde{t}}_i)}{(1-\sqrt{\mu _i(1-\phi _i)})^2}\right\} . \end{aligned}$$