Hierarchical clustering with discrete latent variable models and the integrated classification likelihood

Abstract

Finding a set of nested partitions of a dataset is useful to uncover relevant structure at different scales, and is often dealt with a data-dependent methodology. In this paper, we introduce a general two-step methodology for model-based hierarchical clustering. Considering the integrated classification likelihood criterion as an objective function, this work applies to every discrete latent variable models (DLVMs) where this quantity is tractable. The first step of the methodology involves maximizing the criterion with respect to the partition. Addressing the known problem of sub-optimal local maxima found by greedy hill climbing heuristics, we introduce a new hybrid algorithm based on a genetic algorithm which allows to efficiently explore the space of solutions. The resulting algorithm carefully combines and merges different solutions, and allows the joint inference of the number K of clusters as well as the clusters themselves. Starting from this natural partition, the second step of the methodology is based on a bottom-up greedy procedure to extract a hierarchy of clusters. In a Bayesian context, this is achieved by considering the Dirichlet cluster proportion prior parameter $\alpha$ as a regularization term controlling the granularity of the clustering. A new approximation of the criterion is derived as a log-linear function of $\alpha$, enabling a simple functional form of the merge decision criterion. This second step allows the exploration of the clustering at coarser scales. The proposed approach is compared with existing strategies on simulated as well as real settings, and its results are shown to be particularly relevant. A reference implementation of this work is available in the R-package greed accompanying the paper.

A Dealing with bipartitions

Genetic algorithm The hybrid algorithm presented in Sect. 2 can be easily extended to the co-clustering problem. The latter simultaneously seeks for a partition of the n rows and p columns of a data matrix $\varvec{X}\in {\mathbb {R}}^{n\times p}$. In this case, we work with a partition $\mathcal {P}$ of $\{1,\ldots , n+p\}$ with the additional constraints that it decomposes into two disjoint sets of clusters that corresponds to a partition of $\{1,\ldots ,n\}$ and $\{n+1,\ldots ,n+p\}$ respectively (one for the rows and one for the columns):

$$\begin{aligned} \mathcal {P} =\left\{ C_1^r,\ldots ,C_{{K_{r}}}^r,C_1^c,\ldots ,C_{{K_{c}}}^c\right\} :\left\{ \begin{array}{cl} \bigcup _{k}C^r_k &{}= \{1,\ldots ,n\},\\ \bigcup _{l}C^c_l &{}= \{n+1,\ldots ,n+p\} \end{array} \right. . \end{aligned}$$

(15)

This constraint can be easily incorporated by defining ${{\,\mathrm{ICL_{\textit{ex}}}\,}}(\mathcal {P})=-\infty$ for partitions that do not fulfill this constraint and by initializing the algorithm with admissible solutions. This is sufficient to ensure that the obtained solutions will also be compatible with the constraints, since the admissible set of partitions is closed under the crossover and mutation operations used by the algorithm.

Hierarchical algorithm Furthermore, the hierarchical methodology of Sect. 3 can also be easily extended to bi-partitions. Indeed, the LBM prior

$$\begin{aligned} p(\varvec{\pi }\mid \alpha ) = {{\,\mathrm{Dir}\,}}_{{K_{r}}}(\varvec{\pi }^{r}\mid \alpha ) \times {{\,\mathrm{Dir}\,}}_{{K_{c}}}(\varvec{\pi }^{c}\mid \alpha ), \end{aligned}$$

leaves a factorized integrated likelihood for $p(\varvec{Z}\mid \alpha )$, with a common parameter $\alpha$ (see Côme et al. 2021 for a detailed discussion). Thus, the ${{\,\mathrm{ICL_{\textit{lin}}}\,}}$ approximation of Equation (10) is still log-linear in $\alpha$ and writes:

$$\begin{aligned} {{\,\mathrm{ICL_{\textit{lin}}}\,}}(\varvec{Z}, \alpha ) = ({K_{r}}-1) \log (\alpha ) + ({K_{c}}- 1) \log (\alpha ) + I(\varvec{Z}), \end{aligned}$$

(16)

with $I(\varvec{Z}) = I(\varvec{Z}^{r}) + I(\varvec{Z}^{c})$ the intercepts defined in Equation (9). Hence, with the constraint that a merge cannot be done between rows and columns clusters, one can look for the best row or column fusion to do at each step, therefore building two dendrograms in parallel, with a shared ($\alpha _f)_f$ sequence.