Gaussian parsimonious clustering models with covariates and a noise component

Abstract

We consider model-based clustering methods for continuous, correlated data that account for external information available in the presence of mixed-type fixed covariates by proposing the MoEClust suite of models. These models allow different subsets of covariates to influence the component weights and/or component densities by modelling the parameters of the mixture as functions of the covariates. A familiar range of constrained eigen-decomposition parameterisations of the component covariance matrices are also accommodated. This paper thus addresses the equivalent aims of including covariates in Gaussian parsimonious clustering models and incorporating parsimonious covariance structures into all special cases of the Gaussian mixture of experts framework. The MoEClust models demonstrate significant improvement from both perspectives in applications to both univariate and multivariate data sets. Novel extensions to include a uniform noise component for capturing outliers and to address initialisation of the EM algorithm, model selection, and the visualisation of results are also proposed.

Appendices

Appendix A: $\hbox {CO}_{2}$ data: code examples

Code to reproduce both the exhaustive (Listing 1) and greedy forward stepwise (Listing 2) searches for the CO₂ data described in Sect. 5.1, using the MoEClustR package (Murphy and Murphy 2019), is provided below. The code in Listing 1 can be used to reproduce the results in Table 2.

Appendix B: $\hbox {CO}_{2}$ data: initialisation

The solutions for the optimal $G=3$ equal mixing proportion expert network MoEClust model with equal component variances and the explanatory variable ‘GNP’ fit to the CO₂ data with and without the initial partition being passed through Algorithm 1 are shown in Fig. 6. A BIC value of $-155.20$ is achieved after 18 EM iterations with our proposed initialisation strategy compared to a value of $-161.06$ in 30 EM iterations without. While the models differ only in terms of the initialisation strategy employed, Table 2 shows that the model would not have been identified as optimal according to the BIC criterion had Algorithm 1 not been used. The superior solution in Fig. 6a has one cluster with a steep slope and two clusters with near-zero slopes but different intercepts.

Fig. 6

Scatter plots of GNP against CO₂ emissions for $n=28$ countries showing three linear regression components from the optimal MoEClust model, with equal variances and mixing proportions, with (a) and without (b) the initialisation strategy described in Algorithm 1 invoked

Appendix C: AIS data: stepwise model search

For the AIS data, Table 8 gives the results of the greedy forward stepwise model selection strategy described in Algorithm 2, showing the action yielding the best improvement in terms of BIC for each step. This forward search is less computationally onerous than its equivalent in the backwards direction. A 2-component EVEequal mixing proportion expert network MoE model is chosen, in which the mixing proportions are constrained to be equal and Sex enters the expert network. This same model was identified after an exhaustive search over a range of G values, the full range of GPCM covariance constraints, and every possible combination of the BMI and Sex covariates in the gating and expert networks (see Table 5). Note, however, that the remaining covariates in Table 4 are also considered for inclusion here.

Table 8 Results of the forward stepwise model selection algorithm applied to the AIS data where candidate models do not include a noise component

Table 9 Results of the forward stepwise model selection algorithm applied to the AIS data where all candidate models explicitly include a noise component

To give consideration to outlying observations departing from the prevailing pattern of Gaussianity, a separate stepwise search is conducted, starting from a $G=0$ noise-only model, with all candidate models having an additional noise component. Thus, a distinction is made between the model found by following the steps shown in Table 8 with $G=2$EVE Gaussian components, and the model found by the second stepwise search described in Table 9 with three, of which two are EEE Gaussian and one is uniform. Ultimately, the model with the noise component identified in Table 9 is chosen, based on its superior BIC. Aside from the noise component, it similarly includes ‘Sex’ in the expert network, but differs in its treatment of the gating network and the GPCM constraints employed for the Gaussian clusters. It is a full MoE model where the Gaussian clusters have equal volume, shape, and orientation, the expert network includes the covariate ‘Sex’, and the both ‘SSF’ and ‘Ht’ influence the probability of belonging to the Gaussian clusters but not the additional noise component, as per (8).