A two stages sparse SVM training

Abstract

The small number of support vectors is an important factor for SVM to fast deal with very large scale problems. This paper considers fitting each class of data with a plane by a new model, which captures separability information between classes and can be solved by fast core set methods. Then training on the core sets of the fitting-planes yields a very sparse SVM classifier. The computing complexity of the proposed algorithm is up bounded by ${\text{\rm O}}(1/\varepsilon )$. Experimental results show that the new algorithm trains faster than both CVM and SVMperf averagely, and with comparable generalization performance.

Appendix

Proof of Theorem 1

From the definition of $\bar{K}$, given any vector $\bar{\alpha } = \left[ {\begin{array}{*{20}c} {\alpha^{T} } & {\alpha^{*T} } \\ \end{array} } \right]^{T} \ne 0$, and $C^{\prime} = {\raise0.7ex\hbox{${m_{1} C_{n} }$} \!\mathord{\left/ {\vphantom {{m_{1} C_{n} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}$, one has

$$ \begin{aligned} \bar{\alpha }^{T} \bar{K}\bar{\alpha } = & \left[ {\begin{array}{*{20}c} {\alpha^{T} } & {\alpha^{*T} } \\ \end{array} } \right]\bar{K}\left[ {\begin{array}{*{20}c} {\alpha^{T} } & {\alpha^{*T} } \\ \end{array} } \right]^{T} \\ = & \left[ {\begin{array}{*{20}c} {\alpha^{T} } & {\alpha^{*T} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {K_{AA} \alpha + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}E_{AA} \alpha + C^{\prime}I_{AA} \alpha - K_{AA} \alpha^{*} - {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}E_{AA} \alpha^{*} + C^{\prime}I_{AA} \alpha^{*} } \\ { - K_{AA} \alpha - {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}E_{AA} \alpha + C^{\prime}I_{AA} \alpha + K_{AA} \alpha^{*} + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}E_{AA} \alpha^{*} + C^{\prime}I_{AA} \alpha^{*} } \\ \end{array} } \right] \\ = & \alpha^{T} K_{AA} \alpha + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}\alpha^{T} E_{AA} \alpha + C^{\prime}\alpha^{T} I_{AA} \alpha \\ & \quad - 2\alpha^{*T} K_{AA} \alpha - \alpha^{*T} E_{AA} \alpha + 2C^{\prime}\alpha^{*T} I_{AA} \alpha \\ & \quad + \alpha^{*T} K_{AA} \alpha^{*} + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}\alpha^{*T} E_{AA} \alpha^{*} + C^{\prime}\alpha^{*T} I_{AA} \alpha^{*} \\ & = (\alpha^{T} - \alpha^{*T} )K_{AA} (\alpha - \alpha^{*} ) + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}}(\alpha^{T} - \alpha^{*T} )E_{AA} (\alpha - \alpha^{*} ) \\ & \quad + C^{\prime}(\alpha^{T} + \alpha^{*T} )I_{AA} (\alpha + \alpha^{*} ) \\ \end{aligned} $$

Because matrix $E_{AA}$ is positive semidefinite and I _AA is positive definite, so, If K _AA is positive semidefinite, then $\bar{K}$ is positive semidefinite. Furthermore, for the new learning algorithm, $\alpha \ne - \alpha^{*} , \alpha \succ 0, \alpha^{*} \succ 0$, so, $\bar{\alpha }^{\text{T}} \bar{K}\bar{\alpha }$ is always positive. Thus we say $\bar{K}$ is actually strict positive definite in the feasible set. $\square$

Proof of Theorem 2

For the CCMEB of (8), like (10), one has

$$ R = \sqrt { - \bar{\alpha }^{T} \bar{K}\bar{\alpha } + \bar{\alpha }^{T} (diag(\bar{K}) + \Updelta )} ,\quad c = \sum\limits_{i = 1}^{{2m_{1} }} {\bar{\alpha }_{i} } \bar{\varphi }(x_{i} ) $$

Combining it with (16) and the definition of Δ yields

$$ R^{2} = 2\rho_{ + } /C + \left\| c \right\|^{2} + \eta $$

(20)

From (13), (15), (20), (19) and the definition of $\bar{K}$, the squared distance between the center and any point $\bar{\varphi }(x_{j} )$ is

$$ \begin{aligned} (dist(j))^{2} = & \left\| c \right\|^{2} - 2(\bar{K}\bar{\alpha })_{j} + \eta + \left[ {\begin{array}{*{20}c} { - 2\bar{p}} \\ {2\bar{p}} \\ \end{array} } \right]_{j} \\ = & \left\{ {\begin{array}{*{20}c} {\left\| c \right\|^{2} - 2(\bar{K}\bar{\alpha })_{j} + \eta - 2\bar{p}_{j} , j \prec m_{1} } \\ {\left\| c \right\|^{2} - 2(\bar{K}\bar{\alpha })_{j} + \eta + 2\bar{p}_{j} , m_{1} \le j \prec 2m_{1} } \\ \end{array} } \right. \\ \end{aligned} $$

For $j \prec m_{1}$,

$$ \begin{aligned} \left\| c \right\|^{2} - 2(\bar{K}\bar{\alpha })_{j} + \eta - 2\bar{p}_{j} = & \left\| c \right\|^{2} + \eta - 2\bar{p}_{j} - 2(\bar{K}\bar{\alpha })_{j} \\ = & \left\| c \right\|^{2} + \eta - 2\bar{p}_{j} - 2\left( {\sum\limits_{i = 0}^{{m_{1} }} {\left\langle {\bar{\varphi }(x_{i} ),\bar{\varphi }(x_{j} )} \right\rangle } \bar{\alpha }_{i} } \right. \\ & \quad + \left. {\sum\limits_{{i = m_{1} }}^{{2m_{1} }} {\left\langle {\bar{\varphi }(x_{i} ),\bar{\varphi }(x_{j} )} \right\rangle } \bar{\alpha }_{i} } \right) \\ & = \left\| c \right\|^{2} + \eta - 2\bar{p}_{j} - 2\left( {\sum\limits_{i = 0}^{{m_{1} }} {\left\langle {\varphi (x_{i} ),\varphi (x_{j} )} \right\rangle } + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}} + \delta_{i} \delta_{j} } \right)\bar{\alpha }_{i} \\ & \quad + \sum\limits_{{i = m_{1} }}^{{2m_{1} }} {\left( { - \left\langle {\varphi (x_{{i - m_{1} }} ),\varphi (x_{j} ) - {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 2}}\right.\kern-0pt} \!\lower0.7ex\hbox{$2$}} + \delta_{i} \delta_{j} } \right\rangle } \right)\bar{\alpha }_{i} + {\raise0.7ex\hbox{${C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }_{{m_{1} + j}} )} \\ & = \left\| c \right\|^{2} + \eta - 2\left( {{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}( - w_{ + }^{T} \varphi (x_{j} ) - b_{ + } ) + {\raise0.7ex\hbox{${C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} )} \right) \\ & = \left\| c \right\|^{2} + \eta + {\raise0.7ex\hbox{$2$} \!\mathord{\left/ {\vphantom {2 C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(w_{ + }^{T} \varphi (x_{j} ) + b_{ + } ) - {\raise0.7ex\hbox{${2C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{2C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) \\ & = R^{2} - 2\rho_{ + } /C + {\raise0.7ex\hbox{$2$} \!\mathord{\left/ {\vphantom {2 C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(w_{ + }^{T} \varphi (x_{j} ) + b_{ + } ) - {\raise0.7ex\hbox{${2C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{2C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) \\ \end{aligned} $$

Likewise, for $m_{1} \le j \prec 2m_{1}$,

$$ \left\| c \right\|^{2} - 2(\bar{K}\bar{\alpha })_{j} + \eta + 2\bar{p}_{j} = R^{2} - 2\rho_{ + } /C - {\raise0.7ex\hbox{$2$} \!\mathord{\left/ {\vphantom {2 C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(w_{ + }^{T} \varphi (x_{j} ) + b_{ + } ) - {\raise0.7ex\hbox{${2C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{2C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) $$

So, for each point $x_{i} \in CS_{P}^{t}$ with nonzero Lagrange multiplier, one has

if $j \prec m_{1}$, then

$$ \begin{aligned} R^{2} = & R^{2} - 2\rho_{ + } /C + {\raise0.7ex\hbox{$2$} \!\mathord{\left/ {\vphantom {2 C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(w_{ + }^{T} \varphi (x_{j} ) + b_{ + } ) - {\raise0.7ex\hbox{${2C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{2C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) \\ & \quad \Rightarrow w_{ + }^{T} \varphi (x_{j} ) + b_{ + } = \rho_{ + } + C_{n} m_{1} (\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) \\ & \quad \therefore \quad w_{ + }^{T} \varphi (x_{j} ) + b_{ + } \succ \rho_{ + } \\ \end{aligned} $$

if $m_{1} \le j \prec 2m_{1}$, then

$$ \begin{aligned} R^{2} = & R^{2} - 2\rho_{ + } /C - {\raise0.7ex\hbox{$2$} \!\mathord{\left/ {\vphantom {2 C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(w_{ + }^{T} \varphi (x_{j} ) + b_{ + } ) - {\raise0.7ex\hbox{${2C_{n} m_{1} }$} \!\mathord{\left/ {\vphantom {{2C_{n} m_{1} } C}}\right.\kern-0pt} \!\lower0.7ex\hbox{$C$}}(\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) \\ & \quad \Rightarrow - (w_{ + }^{T} \varphi (x_{j} ) + b_{ + } ) = \rho_{ + } + C_{n} m_{1} (\bar{\alpha }_{j} + \bar{\alpha }^{*}_{j} ) \\ & \quad \therefore \quad w_{ + }^{T} \varphi (x_{j} ) + b_{ + } \prec - \rho_{ + } \\ \end{aligned} $$

Thus the point $x_{i} \in CS_{P}^{t}$ with nonzero Lagrange multiplier must lie out of the slab.

One can also prove, in the same way, that each point $x_{i} \in CS_{P}^{t}$ on the ball $B(c^{t} ,r^{t} )$ with zero Lagrange multiplier must lie on bounding planes of the slab; each point $x_{i} \in CS_{P}^{t}$ inside the ball $B(c^{t} ,r^{t} )$ must lie inside the slab. Details are deleted to save space. $\square$

Proof of Theorem 3

Similar to the proof of theorem 2. $\square$

Proof of Theorem 4

From (6) and (7), we can easily have ${\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 {m_{1} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${m_{1} }$}}\xi_{ + }^{T} e_{{m_{1} }} = C_{n}$. So the parameter C _n represents the average extent each point out of the slab.$\square$