Four algorithms to solve symmetric multi-type non-negative matrix tri-factorization problem

Abstract

In this paper, we consider the symmetric multi-type non-negative matrix tri-factorization problem (SNMTF), which attempts to factorize several symmetric non-negative matrices simultaneously. This can be considered as a generalization of the classical non-negative matrix tri-factorization problem and includes a non-convex objective function which is a multivariate sixth degree polynomial and a has convex feasibility set. It has a special importance in data science, since it serves as a mathematical model for the fusion of different data sources in data clustering. We develop four methods to solve the SNMTF. They are based on four theoretical approaches known from the literature: the fixed point method (FPM), the block-coordinate descent with projected gradient (BCD), the gradient method with exact line search (GM-ELS) and the adaptive moment estimation method (ADAM). For each of these methods we offer a software implementation: for the former two methods we use Matlab and for the latter Python with the TensorFlow library. We test these methods on three data-sets: the synthetic data-set we generated, while the others represent real-life similarities between different objects. Extensive numerical results show that with sufficient computing time all four methods perform satisfactorily and ADAM most often yields the best mean square error (MSE). However, if the computation time is limited, FPM gives the best MSE because it shows the fastest convergence at the beginning. All data-sets and codes are publicly available on our GitLab profile.

Appendix

1.1 A calculation of gradient

Here we offer a derivation of (24) and (25) presented in Sect. 4.1. We use notation $[X]_{\mu \nu }$ for component of matrix X in row $\mu$ and column $\nu$ and $\delta _{ij}$ for the Kronecker delta. Objective function being differentiated is

$$\begin{aligned}&\mathrm {SE}= \sum _{i=1}^N \Vert R_i - f({\tilde{G}})f({\tilde{S}}_i)f({\tilde{G}})^\top \Vert ^2 = \sum _{i=1}^N \Vert Z_i\Vert ^2 \\&\quad = \sum _{i=1}^N \sum _{\mu ,\nu } \left[ Z_i\right] _{\mu \nu }^2, \\ \end{aligned}$$

where $Z_i$ is defined in (23). Let us differentiate $\mathrm {SE}$ with respect to a single component of matrix ${\tilde{S}}_i$.

$$\begin{aligned} \frac{\partial \mathrm {SE}}{\partial [{\tilde{S}}_i]_{\rho \sigma }}&= \sum _{i=1}^N \sum _{\mu ,\nu }\frac{\partial [Z_i]_{\mu \nu }^2}{\partial [{\tilde{S}}_i]_{\rho \sigma }} = -2\sum _{i=1}^N \sum _{\mu ,\nu }[Z_i]_{\mu \nu }\frac{\partial \left[ f({\tilde{G}})f({\tilde{S}}_i)f({\tilde{G}})^\top \right] _{\mu \nu }}{\partial [{\tilde{S}}_i]_{\rho \sigma }} \\&= -2\sum _{i=1}^N\sum _{\mu ,\nu }[Z_i]_{\mu \nu }\frac{\partial }{\partial [{\tilde{S}}_i]_{\rho \sigma }} \sum _{p,r}f\left( [{\tilde{G}}]_{\mu p}\right) f\left( [{\tilde{S}}_i]_{pr}\right) f\left( [{\tilde{G}}]_{\nu r}\right) \\&= -2\sum _{i=1}^N\sum _{\mu ,\nu }[Z_i]_{\mu \nu } \sum _{p,r}f\left( [{\tilde{G}}]_{\mu p}\right) f\left( [{\tilde{G}}]_{\nu r}\right) f'\left( [{\tilde{S}}_i]_{pr}\right) \delta _{\rho p}\delta _{\sigma r} \\&= -2\sum _{i=1}^N\sum _{\mu ,\nu }[Z_i]_{\mu \nu } f\left( [{\tilde{G}}]_{\mu \rho }\right) f\left( [{\tilde{G}}]_{\nu \sigma }\right) f'\left( [{\tilde{S}}_i]_{\rho \sigma }\right) \\&= -2\sum _{i=1}^Nf'\left( [{\tilde{S}}_i]_{\rho \sigma }\right) \left[ f({\tilde{G}})^TZ_if({\tilde{G}})\right] _{\rho \sigma } \end{aligned}$$

This result can be written more compactly as

$$\begin{aligned} \varDelta S_i~=~\nabla _{{\tilde{S}}_i}\mathrm {SE}= -2\sum _{i=1}^Nf'({\tilde{S}}_i)\odot \left( f({\tilde{G}})^TZ_if({\tilde{G}})\right) , \end{aligned}$$

which proves (24).

Using the same steps for ${\tilde{G}}$, we obtain (25).

1.2 B Calculation of polynomial used in the exact line search

Here we show exactly how the polynomial (30) used by GM-ELS is calculated. This polynomial tells how $\mathrm {SE}$ changes with respect to step size t when making a move in direction of gradient. For sake of brevity we use notation $X^2=X\odot X$.

$$\begin{aligned} p(t)=\sum _{i=1}^N\Vert R_i-({\tilde{G}}+t\varDelta {\tilde{G}})^2({\tilde{S}}_i+t\varDelta {\tilde{S}})^2({\tilde{G}}+t\varDelta {\tilde{G}})^{2\top }\Vert ^2 =\sum _{j=0}^{12} c_jt^j \end{aligned}$$

To calculate coefficients $c_j$ we first express how matrix $Z_i$ changes with respect to t.

$$\begin{aligned} Z_i(t)=R_i-({\tilde{G}}+t\varDelta {\tilde{G}})^2({\tilde{S}}_i+t\varDelta {\tilde{S}}_i)^2({\tilde{G}}+t\varDelta {\tilde{G}})^{2\top }=\sum _{j=0}^6 A_{ij}t^j, \end{aligned}$$

where matrices $A_{ij}$ can be calculated using simple expansion steps and are equal to

$$\begin{aligned} A_{i0}&=R_i-{\tilde{G}}^2{\tilde{S}}_i^2{\tilde{G}}^{2\top }\\ A_{i1}&=-2{\tilde{G}}^2{\tilde{S}}_i^2({\tilde{G}}\odot \varDelta {\tilde{G}})^\top -2{\tilde{G}}^2({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i){\tilde{G}}^{2\top }-2({\tilde{G}}\odot \varDelta {\tilde{G}}){\tilde{S}}_i^2{\tilde{G}}^{2\top }\\ A_{i2}&=-{\tilde{G}}^2{\tilde{S}}_i^2\varDelta {\tilde{G}}^{2\top }-4{\tilde{G}}^2({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i)({\tilde{G}}\odot \varDelta {\tilde{G}})^\top -{\tilde{G}}^2\varDelta {\tilde{S}}_i^2{\tilde{G}}^{2\top }-\varDelta {\tilde{G}}^2{\tilde{S}}_i^2{\tilde{G}}^{2\top }\\&\quad -4({\tilde{G}}\odot \varDelta {\tilde{G}})({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i){\tilde{G}}^{2\top }-4({\tilde{G}}\odot \varDelta {\tilde{G}}){\tilde{S}}_i^2({\tilde{G}}\odot \varDelta {\tilde{G}})^\top \\ A_{i3}&=-2{\tilde{G}}^2({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i)\varDelta {\tilde{G}}^{2\top }-2{\tilde{G}}^2\varDelta {\tilde{S}}_i^2({\tilde{G}}\odot \varDelta {\tilde{G}})^\top -2({\tilde{G}}\odot \varDelta {\tilde{G}}){\tilde{S}}_i^2\varDelta {\tilde{G}}^{2\top }\\&\quad -8({\tilde{G}}\odot \varDelta {\tilde{G}})({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i)({\tilde{G}}\odot \varDelta {\tilde{G}})^\top -2({\tilde{G}}\odot \varDelta {\tilde{G}})\varDelta {\tilde{S}}_i^2{\tilde{G}}^{2\top }\\&\quad -2\varDelta {\tilde{G}}^2{\tilde{S}}_i^2({\tilde{G}}\odot \varDelta {\tilde{G}})^\top -2\varDelta {\tilde{G}}^2({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i){\tilde{G}}^{2\top }\\ A_{i4}&=-{\tilde{G}}^2\varDelta {\tilde{S}}_i^2\varDelta {\tilde{G}}^{2\top }-4({\tilde{G}}\odot \varDelta {\tilde{G}})({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i)\varDelta {\tilde{G}}^{2\top }-4({\tilde{G}}\odot \varDelta {\tilde{G}})\varDelta {\tilde{S}}_i^2({\tilde{G}}\odot \varDelta {\tilde{G}})^\top \\&\quad -\varDelta {\tilde{G}}^2{\tilde{S}}_i^2\varDelta {\tilde{G}}^{2\top }-4\varDelta {\tilde{G}}^2({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i)({\tilde{G}}\odot \varDelta {\tilde{G}})^\top -\varDelta {\tilde{G}}^2\varDelta {\tilde{S}}_i^2{\tilde{G}}^{2\top }\\ A_{i5}&=-2({\tilde{G}}\odot \varDelta {\tilde{G}})\varDelta {\tilde{S}}_i^2\varDelta {\tilde{G}}^{2\top }-2\varDelta {\tilde{G}}^2({\tilde{S}}_i\odot \varDelta {\tilde{S}}_i)\varDelta {\tilde{G}}^{2\top }-2\varDelta {\tilde{G}}^2\varDelta {\tilde{S}}_i^2({\tilde{G}}\odot \varDelta {\tilde{G}})^\top \\ A_{i6}&=-\varDelta {\tilde{G}}^2\varDelta {\tilde{S}}_i^2\varDelta {\tilde{G}}^{2\top }. \end{aligned}$$

Knowing matrices $A_{ij}$, the element-wise square of $Z_i(t)$ is

$$\begin{aligned} Z_i^2(t)=\sum _{j=0}^{12}B_{ij}t^j\\ \end{aligned}$$

with

$$\begin{aligned} B_{ij}=\sum _rA_{ir}\odot A_{i,j-r}. \end{aligned}$$

(35)

equation (35) is an application of equivalence of polynomial multiplication and discrete convolution. Therefore, the sum is over all r values that lead to legal indices for both $A_{ir}$ and $A_{i,j-r}$ which are all r between $\max (0, j-6)$ and $\min (j, 6)$. Remembering

$$\begin{aligned} p(t)=\sum _{i=1}^N\sum _{\mu ,\nu }\left[ Z_i^2(t)\right] _{\mu \nu }=\sum _{j=0}^{12} c_jt^j \end{aligned}$$

we can now express

$$\begin{aligned} c_j=\sum _{i=1}^N\sum _{\mu ,\nu }\left[ B_{ij}\right] _{\mu \nu }. \end{aligned}$$

(36)

Calculation of coefficients $c_j$ requires far higher number of matrix multiplications compared to the number needed for gradient calculation. When implemented in TensorFlow calculation of $c_j$ is approximately 7 times more expensive than gradient calculation.