Semi-supervised projected model-based clustering

Abstract

We present an adaptation of model-based clustering for partially labeled data, that is capable of finding hidden cluster labels. All the originally known and discoverable clusters are represented using localized feature subset selections (subspaces), obtaining clusters unable to be discovered by global feature subset selection. The semi-supervised projected model-based clustering algorithm (SeSProC) also includes a novel model selection approach, using a greedy forward search to estimate the final number of clusters. The quality of SeSProC is assessed using synthetic data, demonstrating its effectiveness, under different data conditions, not only at classifying instances with known labels, but also at discovering completely hidden clusters in different subspaces. Besides, SeSProC also outperforms three related baseline algorithms in most scenarios using synthetic and real data sets.

Appendices

Appendices

1.1 Appendix 1: Basic EM theory

The density function of an instance $\mathbf x _{i}$ is

$$\begin{aligned} p(\mathbf x _{i} \mid \varTheta ) = \sum _{m=1}^K\pi _{m} p(\mathbf x _{i}\mid \varvec{\theta }_{m}). \end{aligned}$$

We can define a binary random variable $\mathbf z _{i} = (z_{i1}, \ldots ,z_{iK})$, with $z_{im} = 1$ if instance $\mathbf x _{i}$ belongs to component $m$ and with all other elements $z_{im^{\prime }} = 0$, $\forall$ $m^{\prime } \ne m$. Besides, $p(z_{im} = 1) = \pi _{m}$. Therefore, we can write

$$\begin{aligned} p(\mathbf z _{i}) =\prod _{m=1}^K \pi _{m}^{z_{im}}. \end{aligned}$$

(10)

Also, $p(\mathbf x _{i}\mid z_{im}=1) = p(\mathbf x _{i}\mid \varvec{\theta }_{m})$, which, extended, is

$$\begin{aligned} p(\mathbf x _{i}\mid \mathbf z _{i}, \varTheta ) =\prod _{m=1}^K p(\mathbf x _{i}\mid \varvec{\theta }_{m})^{z_{im}}. \end{aligned}$$

(11)

Using Eqs. (10) and (11), Eq. (1) is obtained by summing over all possible states of $\mathbf z _{i}$

$$\begin{aligned} p(\mathbf x _{i} \mid \varTheta )&= \sum _\mathbf{z _{i}} p(\mathbf x _{i}, \mathbf z _{i} \mid \varTheta )= \sum _\mathbf{z _{i}} p(\mathbf z _{i})p(\mathbf x _{i}\mid \mathbf z _{i}, \varTheta ) \\&= \sum _\mathbf{z _{i}}\left( \prod _{m=1}^K \pi _{m}^{z_{im}}\prod _{m=1}^K p(\mathbf x _{i}\mid \varvec{\theta }_{m})^{z_{im}}\right) \\&= \sum _{m=1}^K\pi _{m} p(\mathbf x _{i}\mid \varvec{\theta }_{m}). \end{aligned}$$

This mixture of distributions has unknown parameters in $\varTheta$ that must be estimated. These parameters can be obtained using the maximum likelihood estimation method. Therefore, assuming that each instance is independent and identically distributed (i.i.d.), and building the log-likelihood function ($\log L$) from Eq. (1) and extending it to all the instances, we obtain

$$\begin{aligned} \log L(\varTheta \mid \mathcal{X })&= \log p(\mathcal{X }|\varTheta ) \\&= \log \prod _{i=1}^N p(\mathbf x _{i} \mid \varTheta ) \\&= \sum _{i=1}^N\log \left( \sum _{m=1}^K \pi _{m} p(\mathbf x _{i}\mid \varvec{\theta }_{m})\right) . \end{aligned}$$

This log-likelihood function is difficult to maximize because the summation over the components is inside the logarithm function. The log-likelihood function would change if both the latent variables ($\mathcal{Z }$) and the observable data ($\mathcal{X }$) were known. Then, based on Eqs. (10) and (11), we can define the complete-data log-likelihood function as

$$\begin{aligned} \log L(\varTheta \mid \mathcal{X },\mathcal{Z })&= \log p(\mathcal{X },\mathcal{Z }|\varTheta ) \nonumber \\&= \log \prod _{i=1}^N \prod _{m=1}^K \pi _{m}^{z_{im}}p(\mathbf x _{i}\mid \varvec{\theta }_{m})^{z_{im}} \nonumber \\&= \sum _{i=1}^N \sum _{m=1}^K z_{im} \left( \log \pi _{m} + \log p (\mathbf x _{i}\mid \varvec{\theta }_{m}) \right) . \end{aligned}$$

(12)

The maximization of this complete-data log-likelihood function is straightforward because the summation is outside the logarithm. Since the latent variables are unknown we cannot use this function directly. However, we can obtain the expectation of this log-likelihood function with respect to the posterior distribution of the latent variables. This expectation is calculated in iteration $t$, having fixed the parameters from the previous iteration $t-1$, in the E-step of the EM algorithm. After this, the parameters of the distributions are recalculated to maximize this expectation (M-step). These two steps are repeated until a convergence criterion is reached. Hence, the expectation of the complete-data log-likelihood function is given by

$$\begin{aligned} \mathcal{Q }(\varTheta ,\varTheta ^{t-1})&= \mathbb{E }_{\mathcal{Z }\mid \mathcal{X },\varTheta ^{t-1}}[\log L(\varTheta \mid \mathcal{X },\mathcal{Z })] \nonumber \\&= \sum _\mathcal{Z } p(\mathcal{Z }\mid \mathcal{X },\varTheta ^{t-1}) \log p(\mathcal{X },\mathcal{Z }\mid \varTheta ), \end{aligned}$$

(13)

where the posterior distribution of the latent variables given the data and the parameters of the previous iteration $t-1$ using Eq. (12), is

$$\begin{aligned} p(\mathcal{Z }\mid \mathcal{X },\varTheta ^{t-1}) \propto \prod _{i=1}^N \prod _{m=1}^K\left( \pi _{m}p(\mathbf x _{i}\mid \varvec{\theta }_{m}) \right) ^{z_{im}}. \end{aligned}$$

(14)

This factorizes over $i$ so that the $\{ \mathbf z _{i} \}$ in this distribution are independent. Using this posterior distribution and Bayes’ theorem, we can calculate the expected value of each $z_{im}$ (responsibility) as

$$\begin{aligned} \mathbb{E }_{z_{im}\mid \mathbf x _{i},\varvec{\theta }_{m}}[z_{im}]&= \gamma (z_{im}) \\&= \frac{\sum _{z_{im}} z_{im}(\pi _{m}p(\mathbf x _{i}\mid \varvec{\theta }_{m}))^{z_{im}}}{\sum _{z_{im^{\prime }}}(\pi _{m^{\prime }}p(\mathbf x _{i}\mid \varvec{\theta }_{m^{\prime }}))^{z_{im^{\prime }}}} \\&= \frac{\pi _{m}p(\mathbf x _{i}\mid \varvec{\theta }_{m})}{\sum _{m^{\prime }=1}^K\pi _{m^{\prime }}p(\mathbf x _{i}\mid \varvec{\theta }_{m^{\prime }})}\\&= p(z_{im}=1 \mid \mathbf x _{i}, \varvec{\theta }_{m}), \end{aligned}$$

which we can use to calculate the expectation of the complete-data log-likelihood, as

$$\begin{aligned} \mathbb{E }_{\mathcal{Z }\mid \mathcal{X },\varTheta ^{t-1}}[\log L(\varTheta \mid \mathcal{X },\mathcal{Z })] = \sum _{i=1}^N \sum _{m=1}^K \gamma (z_{im}) \left( \log \pi _{m} + \log p (\mathbf x _{i}\mid \varvec{\theta }_{m}) \right) . \end{aligned}$$

1.2 Appendix 2: Including subspaces

Defining, for each component and feature, $\rho _{mj} = p(v_{mj}=1)$, the probability of feature $j$ being relevant to component $m$, the new density function, including the search for subspaces, is

$$\begin{aligned} p(\mathbf x _{i} \mid \varTheta )= \sum _{m=1}^K \pi _{m}\prod _{j=1}^{F}\Bigl (\rho _{mj} p(x_{ij}\mid \theta _{mj})+(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj})\Bigr ). \end{aligned}$$

(15)

To prove this new density function, we can obtain for a component $m$ and an instance $i$,

$$\begin{aligned} p(\mathbf v _{m}\mid z_{im} = 1) = \prod _{j=1}^{F}(\rho _{mj})^{v_{mj}}(1 - \rho _{mj})^{1-v_{mj}}. \end{aligned}$$

This can be extended for all components as

$$\begin{aligned} p(\mathcal{V }\mid \mathbf z _{i}) = \prod _{m=1}^K \left( \prod _{j=1}^{F}(\rho _{mj})^{v_{mj}}(1 - \rho _{mj})^{1-v_{mj}}\right) ^{z_{im}}. \end{aligned}$$

(16)

Besides, we can extend Eq. (11) introducing $\mathcal{V }$, as

$$\begin{aligned} p(\mathbf x _{i} \mid \mathcal{V }, \mathbf z _{i}, \varTheta ) = \prod _{m=1}^K \left( \prod _{j=1}^F p(x_{ij}\mid \theta _{mj})^{v_{mj}} p(x_{ij}\mid \lambda _{mj})^{1-v_{mj}}\right) ^{z_{im}}. \end{aligned}$$

(17)

The new density function, based on Eq. (1), and using Eqs. (10), (16), and (17), is,

$$\begin{aligned} p(\mathbf x _{i} \mid \varTheta )&= \sum _\mathbf{z _{i}}\sum _\mathcal{V } p(\mathbf x _{i},\mathcal{V },\mathbf z _{i} \mid \varTheta ) \\&= \sum _\mathbf{z _{i}}\sum _\mathcal{V } p(\mathbf x _{i} \mid \mathcal{V }, \mathbf z _{i}, \varTheta ) p(\mathcal{V } \mid \mathbf z _{i}) p(\mathbf z _{i}). \end{aligned}$$

The summation over $\mathbf z _{i}$ is solved as in Eq. (1), obtaining,

$$\begin{aligned} p(\mathbf x _{i} \mid \varTheta ) \!=\! \sum _\mathcal{V } \Biggl (\! \sum _{m=1}^K \pi _{m}\prod _{j=1}^{F} \Bigr ( [\rho _{mj} p(x_{ij}\mid \theta _{mj})]^{v_{mj}} \!\times \! [(1 \!-\! \rho _{mj}) p(x_{ij}\mid \lambda _{mj})]^{1-v_{mj}}\Bigr ) \Biggr ). \end{aligned}$$

And we can solve the summation over $\mathcal{V }$, summing over all the possible states of each $v_{mj}$, as,

$$\begin{aligned} p(\mathbf x _{i} \mid \varTheta )\!&= \!\sum _{m=1}^K\pi _{m} \prod _{j=1}^{F} \sum _{v_{mj}=0}^1 \Bigl ([\rho _{mj} p(x_{ij}\mid \theta _{mj})]^{v_{mj}} \!\times \! [(1 \!-\! \rho _{mj}) p(x_{ij}\mid \lambda _{mj})]^{1-v_{mj}} \Bigr ) \\&= \sum _{m=1}^K \pi _{m}\prod _{j=1}^{F}\Bigl (\rho _{mj} p(x_{ij}\mid \theta _{mj})+(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj})\Bigr ). \end{aligned}$$

Taking into account that each component can be described in a different feature subspace, this is the new density function of an instance, as shown in Eq. (15).

The new log-likelihood function that should be maximized, by extending Eq. (15) to all the instances, is

$$\begin{aligned} \log L(\varTheta \mid \mathcal{X })&= \log p(\mathcal{X }|\varTheta ) = \log \prod _{i=1}^N p(\mathbf x _{i} \mid \varTheta ) \\&= \sum _{i=1}^N \Biggl ( \log \sum _{m=1}^K \pi _{m}\prod _{j=1}^{F}\Bigl (\rho _{mj} p(x_{ij}\mid \theta _{mj}) +(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj}) \Bigr ) \Biggr ). \end{aligned}$$

This is again difficult to compute since the summation over the components is inside the logarithm function. This equation would change if we knew the sets of latent variables, $\mathcal{Z }$ and $\mathcal V$. Again by extending Eqs. (10), (16), and (17) to all the data, we can write

$$\begin{aligned} p(\mathcal{X },\mathcal{Z },\mathcal V \mid \varTheta )&= \prod _{i=1}^N \prod _{m=1}^K \left( \prod _{j=1}^F p(x_{ij}\mid \theta _{mj})^{v_{mj}} p(x_{ij}\mid \lambda _{mj})^{1-v_{mj}}\right) ^{z_{im}}\\&\times \prod _{i=1}^N \prod _{m=1}^K \left( \prod _{j=1}^{F}(\rho _{mj})^{v_{mj}}(1 - \rho _{mj})^{1-v_{mj}}\right) ^{z_{im}} \\&\times \prod _{i=1}^N \prod _{m=1}^K \pi _{m}^{z_{im}}, \end{aligned}$$

which can be simplified to

$$\begin{aligned} p(\mathcal{X },\mathcal{Z },\mathcal{V } \mid \varTheta )&= \prod _{i=1}^N \prod _{m=1}^K \Biggl ( \pi _{m}^{z_{im}} \prod _{j=1}^F \left( [\rho _{mj} p(x_{ij}\mid \theta _{mj})]^{v_{mj}} \right. \nonumber \\&\times \left. [(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj})]^{1-v_{mj}}\right) ^{z_{im}} \Biggr ). \end{aligned}$$

(18)

We can obtain the complete-data log-likelihood function by taking the logarithm of the previous function as,

$$\begin{aligned} \log L(\varTheta \mid \mathcal{X },\mathcal{Z },\mathcal{V })&= \log p(\mathcal{X },\mathcal{Z },\mathcal{V }\mid \varTheta ) \\&= \log \prod _{i=1}^N \prod _{m=1}^K \Biggl ( \pi _{m}^{z_{im}} \prod _{j=1}^F \left( [\rho _{mj} p(x_{ij}\mid \theta _{mj})]^{v_{mj}} \right. \\&\times \left. [(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj})]^{1-v_{mj}}\right) ^{z_{im}} \Biggr ), \end{aligned}$$

and operating again,

$$\begin{aligned}&\log L(\varTheta \mid \mathcal{X },\mathcal{Z },\mathcal{V }) \nonumber \\&\quad = \sum _{i=1}^N \sum _{m=1}^K \Bigl ( z_{im}\log \pi _{m}+ \sum _{j=1}^F\left( z_{im} \left[ v_{mj} (\log \rho _{mj} + \log p(x_{ij}\mid \theta _{mj})) \right. \right. \nonumber \\&\qquad + \left. \left. (1-v_{mj})(\log (1 - \rho _{mj}) + \log p(x_{ij}\mid \lambda _{mj}))\right] \right) \Bigr ). \end{aligned}$$

(19)

1.3 Appendix 3: Expectation of the complete-data log-likelihood function

Similarly to Eq. (13), the expectation of the complete-data log-likelihood function can be written as

$$\begin{aligned} \mathbb{E }_{\mathcal{Z },\mathcal{V }\mid \mathcal{X },\varTheta ^{t-1}}[\log L(\varTheta \mid \mathcal{X },\mathcal{Z },\mathcal{V })] \!=\! \sum _\mathcal{Z }\sum _\mathcal{V } p(\mathcal{Z },\mathcal{V } \mid \mathcal{X },\varTheta ^{t-1}) \log p(\mathcal{X },\mathcal{Z },\mathcal{V }\mid \varTheta ). \end{aligned}$$

As in Eq. (14), the posterior distribution of the latent variables given the data, having fixed the parameters of the previous iteration $t-1$, and using Eq. (18), can be written as

$$\begin{aligned} p(\mathcal{Z },\mathcal{V } \mid \mathcal{X },\varTheta ^{t-1})&\propto \prod _{i=1}^N \prod _{m=1}^K \Biggl ( \pi _{m}^{z_{im}} \prod _{j=1}^F \left( [\rho _{mj} p(x_{ij}\mid \theta _{mj})]^{v_{mj}} \right. \\&\times \left. [(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj})]^{1-v_{mj}}\right) ^{z_{im}} \Biggr ). \end{aligned}$$

Before computing the expected values of each $v_{mj}$ and each $z_{im}$, we need to define some other necessary probabilities:

$$\begin{aligned} p(x_{ij},v_{mj}=1 \mid \theta _{mj}) = \rho _{mj}p(x_{ij}\mid \theta _{mj}), \end{aligned}$$

and, similarly

$$\begin{aligned} p(x_{ij}, v_{mj}=0 \mid \theta _{mj}) = (1-\rho _{mj})p(x_{ij}\mid \lambda _{mj}). \end{aligned}$$

Taking both expressions into account, we have

$$\begin{aligned} p(x_{ij} \mid \theta _{mj})&= p(x_{ij},v_{mj}=1 \mid \theta _{mj}) + p(x_{ij}, v_{mj}=0 \mid \theta _{mj}) \\&= \rho _{mj}p(x_{ij}\mid \theta _{mj}) + (1-\rho _{mj})p(x_{ij}\mid \lambda _{mj}). \end{aligned}$$

Now, as detailed after Eq. (14), we can calculate the expected value of each $v_{mj}$, as

$$\begin{aligned}&\mathbb{E }_{v_{mj}, \mid \mathbf x _{ij},\varvec{\theta }_{mj}}[v_{mj}] = \gamma (v_{mj}) \\&\quad =\frac{ \rho _{mj}p(x_{ij}\mid \theta _{mj})}{\rho _{mj}p(x_{ij}\mid \theta _{mj}) + (1-\rho _{mj})p(x_{ij}\mid \lambda _{mj})} \\&\quad =p(v_{mj}=1 \mid x_{ij}, \theta _{mj}). \end{aligned}$$

Using this, we calculate the expected value of each $z_{im}$

$$\begin{aligned}&\mathbb{E }_{z_{im} \mid \mathbf v _{m},\mathbf x _{i},\varvec{\theta }_{m}}[z_{im}] = \gamma (z_{im}) \\&\quad =\frac{\pi _{m} \prod _{j=1}^F[\rho _{mj} p(x_{ij}\mid \theta _{mj})+(1 - \rho _{mj}) p(x_{ij}\mid \lambda _{mj})]}{\sum _{m^{\prime }=1}^K \pi _{m^{\prime }} \prod _{j=1}^F[\rho _{m^{\prime }j} p(x_{ij}\mid \theta _{m^{\prime }j})+(1 - \rho _{m^{\prime }j}) p(x_{ij}\mid \lambda _{m^{\prime }j})]} \\&\quad =p(z_{im} = 1 \mid \mathbf v _m, \mathbf x _{i}, \varvec{\theta }_{m}). \end{aligned}$$

Thus the expectation of the complete-data log-likelihood, as in Eq. (3) and using Eq. (19), is

$$\begin{aligned}&\mathbb{E }_{\mathcal{Z },\mathcal{V }\mid \mathcal{X },\varTheta ^{t-1}}[\log L(\varTheta \mid \mathcal{X },\mathcal{Z },\mathcal{V })]\\&\quad = \sum _{i=1}^N \sum _{m=1}^K \gamma (z_{im}) \\&\qquad \times \Biggl ( \log \pi _{m} + \sum _{j=1}^F \Biggl ( \gamma (v_{mj}) (\log \rho _{mj} + \log p(x_{ij}\mid \theta _{mj})) \\&\qquad + (1- \gamma (v_{mj}))(\log (1 - \rho _{mj}) + \log p(x_{ij}\mid \lambda _{mj})) \Biggr ) \Biggr ). \end{aligned}$$

Then, for simplicity’s sake, we define

$$\begin{aligned} \gamma (u_{imj})&= \gamma (z_{im}) \gamma (v_{mj}),\\ \gamma (w_{imj})&= \gamma (z_{im}) (1-\gamma (v_{mj})). \end{aligned}$$

Now we can obtain the expectation of the complete-data log-likelihood as

$$\begin{aligned}&\mathbb{E }_{\mathcal{Z },\mathcal{V }\mid \mathcal{X },\varTheta ^{t-1}}[\log L(\varTheta \mid \mathcal{X },\mathcal{Z },\mathcal{V })]\\&\quad =\sum _{i=1}^N \sum _{m=1}^K \gamma (z_{im}) \log \pi _{m}\\&\qquad + \sum _{i=1}^N \sum _{m=1}^K\sum _{j=1}^F \gamma (u_{imj})(\log \rho _{mj} + \log p(x_{ij}\mid \theta _{mj})) \\&\qquad +\sum _{i=1}^N \sum _{m=1}^K\sum _{j=1}^F \gamma (w_{imj})(\log (1 - \rho _{mj}) + \log p(x_{ij}\mid \lambda _{mj})). \end{aligned}$$

1.4 Appendix 4: M-step

The parameters are recalculated in the M-step to maximize the value of the expectation of the complete-data log-likelihood function. As already mentioned, these updates are obtained by computing the partial derivatives of this expectation and equaling to zero. The univariate Gaussian distribution for each feature and component is used for this explanation. Therefore $\theta _{mj}$ = ($\mu _{\theta _{mj}}$, $\sigma _{\theta _{mj}}^{2})$, and

$$\begin{aligned} \log p(x_{ij}\mid \theta _{mj}) = \log (\sigma _{\theta _{mj}}^{-1} (2\pi )^{-\frac{1}{2}}) - \frac{1}{2}(x_{ij}-\mu _{\theta _{mj}})^2\sigma _{\theta _{mj}}^{-2}. \end{aligned}$$

All the detailed steps of how to update each parameter, follow.

• $\pi _{m}$ is updated³ using a Lagrange multiplier to enforce constraint $\sum _{m=1}^{C+1}\pi _{m} = 1$:

  $$\begin{aligned}&\frac{{\partial }}{{\partial \pi _{m}}}\left( \sum _{i=1}^L\sum _{m=1}^C z_{im}\log \pi _{m}\right. \\&\quad \qquad \;\;+ \sum _{i=L+1}^N\sum _{m=1}^{C+1}\gamma (z_{im})\log \pi _{m} \\&\quad \qquad \left. \;\;+\, \lambda \left( \sum _{m=1}^{C+1}\pi _{m} -1 \right) \right) = 0, \quad \forall m = 1, \ldots , C+1, \end{aligned}$$

  whose derivative is

  $$\begin{aligned} \sum _{i=1}^L z_{im} \frac{1}{\pi _{m}}+ \sum _{i=L+1}^N \gamma (z_{im}) \frac{1}{\pi _{m}} + \lambda = 0. \end{aligned}$$

  Multiplying both sides by $\pi _{m}$ and summing over $m$, with $m = 1, \ldots , C+1$, we have $\lambda = -N$, as

  $$\begin{aligned} - \lambda = \sum _{i=1}^L\sum _{m=1}^{C+1} z_{im} + \sum _{i=L+1}^N\sum _{m=1}^{C+1} \gamma (z_{im}) = N, \end{aligned}$$

  and then we update each $\pi _{m}$ by using

  $$\begin{aligned} \pi _{m} = \frac{\sum _{i=1}^L z_{im}+\sum _{i=L+1}^N \gamma (z_{im})}{N}. \end{aligned}$$

• $\mu _{\theta _{mj}}$ is updated solving the following partial derivative equation:

  $$\begin{aligned}&\frac{{\partial }}{{\partial \mu _{\theta _{mj}}}}\Biggl ( \sum _{i=1}^L\sum _{m=1}^C \sum _{j=1}^F \gamma (u_{imj})\log p(x_{ij}\mid \theta _{mj}) \\&\qquad \qquad + \sum _{i=L+1}^N \sum _{m=1}^{C+1} \sum _{j=1}^F\gamma (u_{imj})\log p(x_{ij}\mid \theta _{mj}) \Biggr ) =0. \end{aligned}$$

  Then the result is

  $$\begin{aligned}&\sum _{i=1}^L \Bigl ( \gamma (u_{imj})\sigma _{\theta _{mj}}^{-2}x_{ij} - \gamma (u_{imj})\sigma _{\theta _{mj}}^{-2}\mu _{\theta _{mj}} \Bigr ) \\&\quad + \sum _{i=L+1}^N \Bigl (\gamma (u_{imj})\sigma _{\theta _{mj}}^{-2}x_{ij} - \gamma (u_{imj})\sigma _{\theta _{mj}}^{-2}\mu _{\theta _{mj}}\Bigr ) = 0, \end{aligned}$$

  and the value of the parameter can be found as

  $$\begin{aligned} \mu _{\theta _{mj}}&= \frac{\sum _{i=1}^L \gamma (u_{imj})x_{ij} + \sum _{i=L+1}^N \gamma (u_{imj})x_{ij}}{\sum _{i=1}^L \gamma (u_{imj})+ \sum _{i=L+1}^N \gamma (u_{imj})} \\&= \frac{\sum _{i=1}^N \gamma (u_{imj})x_{ij}}{\sum _{i=1}^N \gamma (u_{imj})},\quad \forall m = 1, \ldots , C+1; j = 1,\ldots ,F. \end{aligned}$$

• And for $\sigma _{\theta _{mj}}^{2}$,

  $$\begin{aligned}&\frac{{\partial }}{{\partial \sigma _{\theta _{mj}}^{2}}}\Biggl ( \sum _{i=1}^L\sum _{m=1}^C\sum _{j=1}^F \gamma (u_{imj})\log p(x_{ij}\mid \theta _{mj}) \\&\qquad \qquad + \sum _{i=L+1}^N \sum _{m=1}^{C+1}\sum _{j=1}^F \gamma (u_{imj})\log p(x_{ij}\mid \theta _{mj})\Biggr ) = 0. \end{aligned}$$

  The derivative is

  $$\begin{aligned}&\sum _{i=1}^L \Bigl (\gamma (u_{imj})(x_{ij}-\mu _{mj})^2 \sigma _{\theta _{mj}}^{-4} -\gamma (u_{imj})\sigma _{\theta _{mj}}^{-2}\Bigr ) \\&\quad + \sum _{i=L+1}^N \Bigl (\gamma (u_{imj})(x_{ij}-\mu _{mj})^2 \sigma _{\theta _{mj}}^{-4} -\gamma (u_{imj})\sigma _{\theta _{mj}}^{-2}\Bigr ) = 0, \end{aligned}$$

  and the parameter update is

  $$\begin{aligned} \sigma _{\theta _{mj}}^{2}&= \frac{\sum _{i=1}^L \gamma (u_{imj})(x_{ij}-\mu _{\theta _{mj}})^2+ \sum _{i=L+1}^N \gamma (u_{imj})(x_{ij}-\mu _{\theta _{mj}})^2}{\sum _{i=1}^L \gamma (u_{imj})+ \sum _{i=L+1}^N\gamma (u_{imj})}\\&= \frac{\sum _{i=1}^N \gamma (u_{imj})(x_{ij}-\mu _{\theta _{mj}})^2}{\sum _{i=1}^N\gamma (u_{imj})},\quad \forall m = 1, \ldots , C+1; j = 1,\ldots ,F. \end{aligned}$$

  The same development is valid for $\lambda _{mj} = (\mu _{\lambda _{mj}}$, $\sigma _{\lambda _{mj}}^{2})$ but using $\gamma (w_{imj})$ instead of $\gamma (u_{imj})$ to indicate that feature $j$ is irrelevant for component $m$.

• Then, we update $\mu _{\lambda _{mj}}$ as

  $$\begin{aligned} \mu _{\lambda _{mj}} = \frac{\sum _{i=1}^N\gamma (w_{imj})x_{ij}}{\sum _{i=1}^N \gamma (w_{imj})}, \quad \forall m = 1, \ldots , C+1; j = 1,\ldots ,F. \end{aligned}$$

• And for $\sigma _{\lambda _{mj}}^{2}$

  $$\begin{aligned} \sigma _{\lambda _{mj}}^{2} = \frac{\sum _{i=1}^N \gamma (w_{imj})(x_{ij}-\mu _{\lambda _{mj}})^2}{\sum _{i=1}^N \gamma (w_{imj})}, \quad \forall m = 1, \ldots , C+1; j = 1,\ldots ,F. \end{aligned}$$

• Finally in $\mathcal{M }^1$, $\rho _{mj}$ is updated by

  $$\begin{aligned}&\frac{{\partial }}{{\partial \rho _{mj}}} \Biggl ( \sum _{i=1}^L\sum _{m=1}^C \sum _{j=1}^F \gamma (u_{imj})\log \rho _{mj}\\&\qquad \quad \;\;+ \sum _{i=L+1}^N \sum _{m=1}^{C+1}\sum _{j=1}^F \gamma (u_{imj})\log \rho _{mj} \\&\qquad \quad \;\;+\sum _{i=1}^L \sum _{m=1}^C \sum _{j=1}^F \gamma (w_{imj})\log (1- \rho _{mj}) \\&\qquad \quad \;\;+ \sum _{i=L+1}^N \sum _{m=1}^{C+1}\sum _{j=1}^F \gamma (w_{imj})\log (1- \rho _{mj}) \Biggr )= 0, \end{aligned}$$

  whose partial derivative solution is,

  $$\begin{aligned} \sum _{i=1}^N \gamma (u_{imj})\frac{1}{\rho _{mj}} - \sum _{i=1}^N \gamma (w_{imj}) \frac{1}{1-\rho _{mj}}=0. \end{aligned}$$

  This parameter is updated by

  $$\begin{aligned} \rho _{mj} = \frac{\sum _{i=1}^N \gamma (u_{imj})}{\sum _{i=1}^L z_{im} + \sum _{i=L+1}^N \gamma (z_{im})},\quad \forall m = 1, \ldots , C+1; j = 1,\ldots ,F. \end{aligned}$$

  Note that $z_{i,C+1} = 0$ for $i = 1,\ldots ,L$ for the three sets of parameters, $\theta _{mj}$, $\lambda _{mj}$ and $\rho _{mj}$.