Reduced Analytic Dependency Modeling: Robust Fusion for Visual Recognition

Abstract

This paper addresses the robustness issue of information fusion for visual recognition. Analyzing limitations in existing fusion methods, we discover two key factors affecting the performance and robustness of a fusion model under different data distributions, namely (1) data dependency and (2) fusion assumption on posterior distribution. Considering these two factors, we develop a new framework to model dependency based on probabilistic properties of posteriors without any assumption on the data distribution. Making use of the range characteristics of posteriors, the fusion model is formulated as an analytic function multiplied by a constant with respect to the class label. With the analytic fusion model, we give an equivalent condition to the independent assumption and derive the dependency model from the marginal distribution property. Since the number of terms in the dependency model increases exponentially, the Reduced Analytic Dependency Model (RADM) is proposed based on the convergent property of analytic function. Finally, the optimal coefficients in the RADM are learned by incorporating label information from training data to minimize the empirical classification error under regularized least square criterion, which ensures the discriminative power. Experimental results from robust non-parametric statistical tests show that the proposed RADM method statistically significantly outperforms eight state-of-the-art score-level fusion methods on eight image/video datasets for different tasks of digit, flower, face, human action, object, and consumer video recognition.

Appendices

Appendix A: Proof of Proposition 1

We first show that conditionally independent condition implies the solution to the equation system (16) is trivial, i.e. ${\varvec{a}}_{lm0} = \mathbf 0 , {\varvec{a}}_{lm2} = \mathbf 0 , {\varvec{a}}_{lm3} = \mathbf 0 , \ldots$ is a trivial solution to equation system (16) for $m = 1, \ldots , M$. If feature representations are independent with each other given class label $\omega _l$, the analytic function $h_l({\varvec{s}}_l; {\varvec{a}}_l)$ becomes Eq. (3). Rewriting the analytic function in (3) according to the order of $s_{lm}$, we get

$$\begin{aligned} h_l({\varvec{s}}_l; {\varvec{a}}_l) = g_{lm1}(\tilde{{\varvec{s}}}_{lm}; {\varvec{a}}_{lm1}) s_{lm} \end{aligned}$$

(36)

where $g_{lm1}(\tilde{{\varvec{s}}}_{lm}; {\varvec{a}}_{lm1}) = p_l^{1-M} \prod _{m' \ne m} s_{lm'}$. This Eq. (36) means that $g_{lmn}(\tilde{{\varvec{s}}}_{lm}; {\varvec{a}}_{lmn}) \equiv 0$ or equivalently ${\varvec{a}}_{lmn} = \mathbf 0$ for $n \ne 1$, i.e. the solution to equation system (16) is trivial.

On the other hand, given the solution to equation system (16) is trivial, we need to show that the analytic function $h_l({\varvec{s}}_l; {\varvec{a}}_l)$ is equal to Eq. (3). If ${\varvec{a}}_{lmn} = \mathbf 0$ for $n \ne 1$, then the analytic function $h_l({\varvec{s}}_l; {\varvec{a}}_l)$ can be rewritten as Eq. (36) for $m = 1, \ldots , M$. This implies each term in the power series $h_l$ contains all variables $s_{l1}, \ldots , s_{lM}$ and the order of each $s_{lm}$ cannot be larger than one. In this case, there is only one non-zero term $\prod _{m=1}^{M} s_{lm}$ in the analytic function $h_l$. In addition, according to the normalization equation (15), the non-zero term $\prod _{m=1}^{M} s_{lm}$ is normalized by the prior. And the analytic function becomes equation (3). This complete the proof of this proposition.

Appendix B: Derivation for $E_{\mathrm{Dis}}({\varvec{a}}, {\varvec{q}})$

$$\begin{aligned}&E_\mathrm{Dis }({\varvec{a}}, {\varvec{q}}) \\&\quad = - \theta \sum _{l=1}^L \sum _{l' \ne l} \sum _{{y_j} = \omega _l } q_{jl'} \\&\qquad +\,\, \frac{\theta }{2} \sum _{l=1}^L \sum _{l' \ne l} \sum _{{y_j} = \omega _l } (({\varvec{a}}_l^T {\varvec{z}}_{jl} - {\varvec{a}}_l'^T {\varvec{z}}_{jl'}) - q_{jl'})^2 \\&\quad = - \theta \sum _{l=1}^L \sum _{l' \ne l} {\varvec{q}}_{ll'}^T \mathbf 1 + \frac{\theta }{2} \sum _{l=1}^L \sum _{l' \ne l} \Vert (Z_{ll}^T {\varvec{a}}_l - Z_{ll'}^T {\varvec{a}}_l') - {\varvec{q}}_{ll'}\Vert ^2 \\&\quad = - \theta \sum _{l=1}^L {\varvec{q}}_l^T \mathbf 1 + \frac{\theta }{2} \sum _{l=1}^L ({\varvec{a}}^T Z_l - {\varvec{q}}_l^T) (Z_l^T {\varvec{a}} - {\varvec{q}}_l) \\&\quad = \frac{1}{2} {\varvec{a}}^T H_\mathrm{Dis } {\varvec{a}} + \theta \sum _{l=1}^L (\frac{1}{2} {\varvec{q}}_l^T {\varvec{q}}_l - {\varvec{a}}^T Z_l {\varvec{q}}_l - {\varvec{q}}_l^T \mathbf 1 ) \end{aligned}$$

Appendix C: Derivation of the Matrix Formulation for $E({\varvec{a}})$

$$\begin{aligned}&E({\varvec{a}}) \\&\quad = \frac{\sum _{l=1}^{L} \sum _{m=1}^M \Vert {\varvec{a}}_l^T ({\varvec{c}}_{lm0}, \ldots , {\varvec{c}}_{lmN}) - (b_0, \ldots , b_N)\Vert ^2}{2LM(N+1)} \\&\quad = \frac{\sum _{l=1}^{L} \sum _{m=1}^M ({\varvec{a}}_l^T C_{lm} - {\varvec{b}}^T) (C_{lm}^T {\varvec{a}}_l - {\varvec{b}})}{2LM(N+1)} \\&\quad = \frac{\sum _{l=1}^{L} [{\varvec{a}}_l^T (\sum _{m=1}^M C_{lm} C_{lm}^T) {\varvec{a}}_l - 2 {\varvec{a}}_l^T \sum _{m=1}^M C_{lm} {\varvec{b}} + {\varvec{b}}^T {\varvec{b}}]}{2LM(N+1)} \\&\quad = \frac{1}{2} \sum _{l=1}^{L} {\varvec{a}}_l^T H_l {\varvec{a}}_l - \sum _{l=1}^{L} {\varvec{a}}_l^T {\varvec{f}}_l + \frac{1}{2LM(N+1)} \sum _{l=1}^{L} {\varvec{b}}^T {\varvec{b}}\\&\quad = \frac{1}{2} {\varvec{a}}^T H {\varvec{a}} - {\varvec{a}}^T {\varvec{f}} + \frac{1}{2LM(N+1)} \sum _{l=1}^{L} {\varvec{b}}^T {\varvec{b}} \end{aligned}$$