Variational discriminant analysis with variable selection

Abstract

A fast Bayesian method that seamlessly fuses classification and hypothesis testing via discriminant analysis is developed. Building upon the original discriminant analysis classifier, modelling components are added to identify discriminative variables. A combination of cake priors and a novel form of variational Bayes we call reverse collapsed variational Bayes gives rise to variable selection that can be directly posed as a multiple hypothesis testing approach using likelihood ratio statistics. Some theoretical arguments are presented showing that Chernoff-consistency (asymptotically zero type I and type II error) is maintained across all hypotheses. We apply our method on some publicly available genomics datasets and show that our method performs well in practice for its computational cost. An R package VaDA has also been made available on Github.

Appendix

1.1 VQDA derivations

In the VQDA setting ($\sigma _{j 1}^2 \ne \sigma _{j 0}^2$) the posterior distribution of $\gamma _j$ given ${\mathcal {D}}$ and $\mathbf{x}_{n+1}$ may be expressed as

$$\begin{aligned} p(\gamma _j \; | \; {\mathcal {D}}, \mathbf{x}_{n+1}) = \frac{ p(\gamma _j, {\mathcal {D}}, \mathbf{x}_{n+1})}{ p({\mathcal {D}}, \mathbf{x}_{n+1}) }. \end{aligned}$$

By letting $h \rightarrow \infty$, the marginal likelihood of the data in the denominator is of the same form as Eq. (9) with the exception that

$$\begin{aligned} {\varvec{\theta }}_1 = ({\varvec{\mu }}_1, {\varvec{\mu }}_0, {\varvec{\mu }}, {\varvec{\sigma }}_1^2, {\varvec{\sigma }}_0^2, {\varvec{\sigma }}^2, \rho _y, \rho _\gamma ), \end{aligned}$$

and

$$\begin{aligned}&\lambda _{\text {LRT}} (\widetilde{\mathbf{x}}_j, x_{n+1,j}, \mathbf{y}, y_{n+1}) \\&\quad = (n+1) \log (\widehat{\sigma }_j^2) - (n_1 + y_{n+1}) \log (\widehat{\sigma }_{j1}^2) \\&\qquad - (n_0 + 1 - y_{n+1}) \log (\widehat{\sigma }_{j0}^2), \end{aligned}$$

where

$$\begin{aligned} \widehat{\sigma }_{j 1}^2 =&\tfrac{1}{n_1+y_{n+1}} \bigg [ ||\mathbf{y}^\mathrm{T}\{\widetilde{\mathbf{x}}_{j} - \widehat{\mu }_{j1}\mathbf{1}\}||^2 \\&\quad + y_{n+1} (x_{n+1,j} - \widehat{\mu }_{j1})^2 \bigg ],\\ \widehat{\sigma }_{j 0}^2 =&\tfrac{1}{n_0 +1 - y_{n+1}} \bigg [ ||(\mathbf{1}- \mathbf{y})^\mathrm{T}\{\widetilde{\mathbf{x}}_{j} - \widehat{\mu }_{j0}\mathbf{1}\}||^2 \\&\quad + (1 - y_{n+1}) (x_{n+1,j} - \widehat{\mu }_{j0})^2 \bigg ], \end{aligned}$$

and the $j{\text {th}}$ entry of ${\varvec{\lambda }}_{\text {Bayes}}$ is (as $h \rightarrow \infty$)

$$\begin{aligned}&\lambda _{\text {Bayes}} (\widetilde{\mathbf{x}}_j, x_{n+1,j}, \mathbf{y}, y_{n+1}) \\&\quad \rightarrow \lambda _{\text {LRT}}(\widetilde{\mathbf{x}}_j, x_{n+1,j}, \mathbf{y}, y_{n+1}) + \log (n_1 + y_{n+1} ) \\&\qquad + \log (n_0 + 1 - y_{n+1} ) - \log (2) - 3\log (n+1) \\&\qquad -\, 2\xi \{(n+1)/2\} + 2\xi \{(n_1 + y_{n+1})/2\} \\&\qquad +\, 2\xi \{ (n_0 + 1 - y_{n+1})/2 \}, \\&= \lambda _{\text {LRT}}(\widetilde{\mathbf{x}}_j, x_{n+1,j}, \mathbf{y}, y_{n+1}) - 2 \log (n+1) \\&\qquad +\,O(n_0^{-1} + n_1^{-1}), \end{aligned}$$

where $\xi (x) = \log \varGamma (x) + x - x \log (x) - \tfrac{1}{2} \log (2 \pi )$. Since the calculation of the marginal likelihood involves a combinatorial sum over $2^{p+1}$ binary combinations, exact Bayesian inference is also computationally impractical in the VQDA setting.

Table 2 Iterative scheme for obtaining the parameters in the optimal densities $q(\varvec{\gamma }, y_{n+1}, \ldots , y_{n+m})$ in VQDA

Similar to VLDA, we will use RCVB to approximate the posterior $p({\varvec{\gamma }}, y_{n+1} | \mathbf{x}, \mathbf{x}_{n+1}, \mathbf{y})$ by

$$\begin{aligned} q(y_{n+1}, {\varvec{\gamma }}) = q (y_{n+1}) \prod _{j=1}^{p} q_j (\gamma _j). \end{aligned}$$

This yields the approximate posterior for $\gamma _j$ as

$$\begin{aligned}&q_j(\gamma _j) \propto \int \exp \big [ {\mathbb {E}}_{-q_j} \{ \log p({\mathcal {D}}, \mathbf{x}_{n+1}, y_{n+1},{\varvec{\gamma }}, {\varvec{\theta }}_1) \} \big ] \text {d} {\varvec{\theta }}_1, \\&\quad \propto \exp \bigg [ {\mathbb {E}}_{-q_j} \Big \{ \log {\mathcal {B}}(a_\gamma + \mathbf{1}^\mathrm{T}{\varvec{\gamma }}, b_\gamma + p - \mathbf{1}^\mathrm{T}{\varvec{\gamma }}) \Big \} \\&\qquad + \tfrac{\gamma _j}{2} {\mathbb {E}}_{-q_j} \Big \{ {\varvec{\lambda }}_{\text {Bayes}} (\widetilde{\mathbf{x}}_j, x_{n+1,j}, \mathbf{y}, y_{n+1}) \Big \} \bigg ]. \end{aligned}$$

For a sufficiently large n, we can avoid the need to evaluate the expectation ${\mathbb {E}}_{-q_j} \Big \{ {\varvec{\lambda }}_{\text {Bayes}} (\widetilde{\mathbf{x}}_j, x_{n+1,j}, \mathbf{y}, y_{n+1}) \Big \}$ by applying Taylor’s expansion to obtain the approximation

$$\begin{aligned}&{\mathbb {E}}_{-q_j} \log (a_\gamma + \mathbf{1}^\mathrm{T} {\varvec{\gamma }}_{-j}) \approx \log (a_\gamma + \mathbf{1}^\mathrm{T} \mathbf{w}_{-j}), \nonumber \\&{\mathbb {E}}_{-q_j} \log (b_\gamma + p - \mathbf{1}^\mathrm{T} {\varvec{\gamma }}_{-j} - 1) \nonumber \\&\quad \approx \log (b_\gamma + p - \mathbf{1}^\mathrm{T} \mathbf{w}_{-j} - 1), \nonumber \\&\widehat{\sigma }_{j 1}^2 \approx \tfrac{1}{n_1} ||\mathbf{y}^\mathrm{T}\{\widetilde{\mathbf{x}}_{j} - \widehat{\mu }_{j1}\mathbf{1}\}||^2, \nonumber \\&\widehat{\sigma }_{j 0}^2 \approx \tfrac{1}{n_0} ||(\mathbf{1}- \mathbf{y})^\mathrm{T}\{\widetilde{\mathbf{x}}_{j} - \widehat{\mu }_{j0}\mathbf{1}\}||^2, \nonumber \\&\widehat{\sigma }_{j}^2 \approx \tfrac{1}{n} ||\widetilde{\mathbf{x}}_{j} - \widehat{\mu }_{j}\mathbf{1}||^2, \nonumber \\&\widehat{\mu }_{j 1} \approx \tfrac{1}{n_1} \mathbf{y}^\mathrm{T} \widetilde{\mathbf{x}}_j , \;\; \widehat{\mu }_{j 0} \approx \tfrac{1}{n_0} (\mathbf{1}- \mathbf{y})^\mathrm{T} \widetilde{\mathbf{x}}_j, \;\; \widehat{\mu }_{j} \approx \tfrac{1}{n} \mathbf{1}^\mathrm{T} \widetilde{\mathbf{x}}_j, \end{aligned}$$

(16)

and, similar to VLDA, ${\varvec{\lambda }}_{\text {Bayes}}$ does not depend on the new observation $(\mathbf{x}_{n+1}, y_{n+1})$. By using the approximation in (16), we have

$$\begin{aligned} w_j&= \frac{q_j(\gamma _j = 1)}{q_j(\gamma _j = 1) + q_j(\gamma _j = 0)}, \\&\approx \text {expit}\bigg [ \log (a_\gamma {+} \mathbf{1}^\mathrm{T} \mathbf{w}_{-j}) {-} \log (b_\gamma + p - \mathbf{1}^\mathrm{T} \mathbf{w}_{-j} - 1) \\&\quad + \tfrac{1}{2} \log (\tfrac{n_1 n_0}{2}) + \xi (\tfrac{n_1}{2}) + \xi (\tfrac{n_0}{2}) - \xi (\tfrac{n}{2}) \\&\quad - \tfrac{3}{2} \log (n+1) + \tfrac{1}{2} \lambda _{\text {LRT}} (\widetilde{\mathbf{x}}_j, y_{n+1}) \bigg ], \\&= \text {expit}\bigg [ \text {penalty}_{QDA,j} + \tfrac{1}{2} \lambda _{\text {LRT}} (\widetilde{\mathbf{x}}_j, y_{n+1}) \bigg ], \end{aligned}$$

To obtain the approximate density for $y_{n+1}$, we integrate analytically over ${\varvec{\theta }}_1$ to obtain

$$\begin{aligned} q(y_{n+1})&\propto \int \exp \big [ {\mathbb {E}}_{-y} \{ \log p({\mathcal {D}}, \mathbf{x}_{n+1}, y_{n+1},{\varvec{\gamma }}, {\varvec{\theta }}_1) \} \big ] \text {d} {\varvec{\theta }}_1, \\&\propto \exp \bigg [ \log {\mathcal {B}}(a_y + n_1 + y_{n+1}, b_y + n_0 + 1 - y_{n+1}) \\&\quad + \mathbf{1}^\mathrm{T} \mathbf{w}\Big \{ \log \varGamma (\tfrac{n_1 + y_{n+1}}{2}) + \log \varGamma (\tfrac{n_0 + 1 - y_{n+1}}{2}) \Big \} \\&\quad + \tfrac{1}{2} \mathbf{w}^\mathrm{T} \Big \{ \log {\varvec{\phi }}(\mathbf{x}_{n+1}; \widehat{{\varvec{\mu }}}_1, \widehat{{\varvec{\sigma }}}_{1}^2) - \log {\varvec{\phi }}(\mathbf{x}_{n+1}; \widehat{{\varvec{\mu }}}_0, \widehat{{\varvec{\sigma }}}_{0}^2) \Big \} \bigg ], \end{aligned}$$

where the $j{\text {th}}$ element of the $p \times 1$ vector ${\varvec{\phi }}(\mathbf{x}_{n+1}; \widehat{{\varvec{\mu }}}_k, \widehat{{\varvec{\sigma }}}_{k}^2)$ is the Gaussian density

$$\begin{aligned} \phi (x_{n+1,j}; \widehat{\mu }_{j k}, \widehat{\sigma }_{j k}^2), \end{aligned}$$

and the $\log$ prefix denotes an element-wise $\log$ of a vector.

In the general case with m new observations, we may apply Taylor’s expansion results from (16) to compute the approximate classification probability for $y_{n+i}$ as

$$\begin{aligned} \widetilde{y}_i&= \frac{q(y_{n+i} = 1)}{q(y_{n+i} = 1) + q(y_{n+i} = 0)}, \\&\approx \text {expit}\bigg [ \log \big ( \tfrac{n_1}{n_0} \big ) + \mathbf{1}^\mathrm{T} \mathbf{w}\big \{ \log \varGamma (\tfrac{n_1 + 1}{2}) - \log \varGamma (\tfrac{n_1}{2}) \\&\quad + \log \varGamma (\tfrac{n_0 + 1}{2}) - \log \varGamma (\tfrac{n_0}{2}) \big \} \\&\quad + \tfrac{1}{2} \mathbf{w}^\mathrm{T} \Big \{ \log {\varvec{\phi }}(\mathbf{x}_{n+i}; \widehat{{\varvec{\mu }}}_1, \widehat{{\varvec{\sigma }}}_{1}^2) - \log {\varvec{\phi }}(\mathbf{x}_{n+i}; \widehat{{\varvec{\mu }}}_0, \widehat{{\varvec{\sigma }}}_{0}^2) \Big \} \bigg ]. \end{aligned}$$

The RCVB algorithm for VQDA may be found in Table 2.