Embedded topics in the stochastic block model

Abstract

Communication networks such as emails or social networks are now ubiquitous and their analysis has become a strategic field. In many applications, the goal is to automatically extract relevant information by looking at the nodes and their connections. Unfortunately, most of the existing methods focus on analysing the presence or absence of edges and textual data is often discarded. However, all communication networks actually come with textual data on the edges. In order to take into account this specificity, we consider in this paper networks for which two nodes are linked if and only if they share textual data. We introduce a deep latent variable model allowing embedded topics to be handled called ETSBM to simultaneously perform clustering on the nodes while modelling the topics used between the different clusters. ETSBM extends both the stochastic block model (SBM) and the embedded topic model (ETM) which are core models for studying networks and corpora, respectively. The inference is done using a variational-Bayes expectation-maximisation algorithm combined with a stochastic gradient descent. The methodology is evaluated on synthetic data and on a real world dataset.

Appendices

Appendix A: Inference

Proof of Proposition 4.1

The ELBO can be decomposed as follow:

$$\begin{aligned}{} & {} \log p(A, W \mid \alpha , \rho ) \\{} & {} \quad = {\mathbb {E}}_{R} \left[ \log p(A, W \mid \alpha , \rho )\right] \\{} & {} \quad = {\mathbb {E}}_{R} \left[ \log \frac{p(A, W, Y, \pi , \gamma , \delta \mid \alpha , \rho )}{p(Y, \pi , \gamma , \delta \mid A, W, \alpha , \rho )} \right] \\{} & {} \qquad \text {applying Bayes rule}\\{} & {} \quad = {\mathbb {E}}_{R} \left[ \log \frac{p(A, W, Y, \pi , \gamma , \delta \mid \alpha , \rho )}{R(Y, \pi , \gamma , \delta )}\right. \\{} & {} \qquad \quad \left. + \log \frac{R(Y, \pi , \gamma , \delta )}{p(Y, \pi , \gamma , \delta \mid A, W, \alpha , \rho )} \right] \\{} & {} \quad = \mathscr {L} (R(\cdot ); \alpha , \rho ) + {{\,\textrm{KL}\,}}(R(\cdot ) || p(Y, \pi , \gamma , \delta \mid A, W, \alpha , \rho )). \end{aligned}$$

$\square$

Proof of Proposition 4.2

$$\begin{aligned}{} & {} \mathscr {L}(R(\cdot ); \alpha , \rho )\nonumber \\{} & {} \quad = \overset{\mathscr {L}^{net}(\tau , \tilde{\pi }_{qr1}, \tilde{\pi }_{qr2} \tilde{\gamma } ; \alpha , \rho ) :=}{\overbrace{{\mathbb {E}}_{R}\left[ \log \frac{p( W \mid Y, A, \theta , \alpha , \rho ) p(\theta )}{ R(\theta )}\right] }}\nonumber \\{} & {} \qquad +\overset{\mathscr {L}^{texts}(\tau , \nu ; \alpha , \rho ) :=}{\overbrace{{\mathbb {E}}_{R} \left[ \log \frac{ p(A \mid Y, \pi ) p(Y \mid \gamma ) p(\pi ) p(\gamma )}{ R(Y) R(\pi ) R(\gamma )} \right] }} \nonumber \\{} & {} \quad = {\mathbb {E}}_{R}\left[ \log p( W \mid Y, A, \theta , \alpha , \rho )\right] +{\mathbb {E}}_{R}\left[ \log p(\theta ) \right] \nonumber \\{} & {} \qquad -{\mathbb {E}}_{R}\left[ \log R(\theta ) \right] \nonumber \\{} & {} \qquad + {\mathbb {E}}_{R}\left[ \log p(A \mid Y, \pi ) \right] +{\mathbb {E}}_{R}\left[ \log p(Y \mid \gamma ) \right] \nonumber \\{} & {} \qquad +{\mathbb {E}}_{R}\left[ \log p(\pi ) \right] + {\mathbb {E}}_{R} \left[ \log p(\gamma ) \right] \nonumber \\{} & {} \qquad - {\mathbb {E}}_{R}\left[ \log R(Y) \right] -{\mathbb {E}}_{R}\left[ \log R(\pi ) \right] - {\mathbb {E}}_{R} \left[ \log R(\gamma ) \right] \nonumber \\{} & {} \quad = \sum _{i \ne j }^M \sum _{ q, r}^Q A_{ij} \tau _{iq} \tau _{jr} {\mathbb {E}}_{R}\left[ \underset{T_{ij}^{\delta _{qr}}}{\underbrace{\log p(w_{ij} \mid \delta _{qr}, \alpha , \rho )}} \right] \nonumber \\{} & {} \qquad -\sum _{q,r} {{\,\textrm{KL}\,}}( {\mathcal {N}}(\mu _{qr}(\tau , \nu ), \sigma _{qr} (\tau ,\nu )) || {\mathcal {N}}(0, I) ) \nonumber \\{} & {} \qquad + \sum _{i \ne j}^M \sum _{q,r}^Q \tau _{iq} \tau _{jr} A_{ij} \left( \psi ( \kappa _{qr1}) - \psi ( \kappa _{qr2}) \right) \nonumber \\{} & {} \qquad +\sum _{i \ne j}^M \sum _{q,r}^Q\tau _{iq} \tau _{jr} (\psi (\kappa _{qr2} ) -\psi (\kappa _{qr1} + \kappa _{qr2} )) \nonumber \\{} & {} \qquad + \sum _{i=1}^M \sum _{q=1}^Q \tau _{iq} \left( \psi (\gamma _{q}) -\psi \left( \sum _{q} \gamma _{q}\right) \right) \nonumber \\{} & {} \qquad + \log {\mathcal {B}}(1_Q) + \log ({\mathcal {B}}(a,b) ) \nonumber \\{} & {} \qquad - \sum _{i=1}^M \sum _{q=1}^Q \tau _{iq} \log (\tau _{iq})\nonumber \\{} & {} \qquad -\sum _{q,r} \log {\mathcal {B}} (\kappa _{qr1}, \kappa _{qr2})-\log {\mathcal {B}}(\gamma ). \end{aligned}$$

(A.1)

where,

$$\begin{aligned} T_{ij}^{\delta _{qr}} = \sum _{d=1}^{D_{ij}} \sum _{n=1}^{N_{id}^d} \sum _{v=1}^V w_{ij}^{dnv} \log \left( \sum _{k=1}^K \theta _{qr k} \beta _{k v} \right) . \end{aligned}$$

(A.2)

and $\theta _{qr} = \mu _{qr}(\tau , \nu ) + \sigma _{qr}(\tau , \nu ) \epsilon$, $\epsilon \sim {\mathcal {N}}(0_K, \textrm{I}_{K})$.

The Kullback–Leibler divergence between two Gaussian variables has a close form and is easy to compute. All the terms can be computed except for the expectation of $T_{ij}^{\delta _{qr}}$ that can be approximated using a Monte-Carlo estimator, by drawing S samples for each pair (q, r), such that:

$$\begin{aligned}&\epsilon ^s \sim {\mathcal {N}}(0,I_{K}), \ \ \ \delta _{qr}^s = \mu _{qr}( \tau , \nu ) +\sigma _{qr}( \tau , \nu ) \odot \epsilon ^s,\\&\theta _{qr}^s = {{\,\textrm{softmax}\,}}(\delta _{qr}^s). \end{aligned}$$

with $\odot$ denoting the Hadamard product. Thus, for each pair of nodes (i, j) and pair of clusters (q, r), the estimate is given by:

$$\begin{aligned} \hat{T}_{ij}^{qr} = S^{-1} \sum _{s=1}^S T_{ij}^{\delta ^s_{qr}}. \end{aligned}$$

Plugging $\hat{T}_{ij}^{qr}$ in the Eq. (A.1) gives the final estimator of the ELBO. $\square$

Fig. 12

The most important words of each topic present in the meta-graph translated in English

Figure 12 provides a translation of topics found by ETSBM on the real dataset and appearing in the meta-network.

Appendix B: Real data

Figure 13 provides a translation of topics found by ETM on the real dataset and appearing in the meta-network.

Fig. 13

The most important words of each topic present in the meta-graph translated in English