Quick Real-Boost with: Weight Trimming, Exponential Impurity, Bins, and Pruning

Abstract

The central point of attention for this paper is weight trimming — a technique known for speeding up boosted learning procedures. The loss of accuracy introduced by the technique is typically negligible. Recently, an elegant algorithm has been proposed by Appel et al.: it applies weight trimming under AdaBoost, prunes some features using a special error bound, but simultanouesly guarantees the same outcome (ensemble of trees with exactly the same parameters) as if with no trimming. Thus, no loss of training accuracy occurs. In this paper, we supplement the idea by Appel with a suitable extension for real-boosting. We prove that this approach gives the same outcome guarantees, both for stumps and trees. Additionally, we analyze the complexity of Appel’s idea and we show that in some cases it may lead to computational losses.

A Proof of Outcome Guarantee for ‘quick’ Tree Growing Procedure with Exponential Impurity

Proof

(Theorem 2 ). First, let us define the Gini error (for the n-subset):

(21)

where $Z_{\rho _l}^{+}=\sum _{i\in \rho _l}w_i[y=+1]$, $Z_{\rho _l}^{-}=\sum _{i\in \rho _l}w_i[y=-1]$ are probability masses for classes. Let us write two representations (22), (23) for the products — mass times Gini error — that will be useful later on:

$$\begin{aligned} Z_{n}\varepsilon _{\mathop {{\scriptscriptstyle {\text {gini}}}}\limits _{n}}&=\sum _l \left( Z_{\rho _l}^{}-\frac{\left( Z_{\rho _l}^{+}\right) ^2}{Z_{\rho _l}^{}}-\frac{\left( Z_{\rho _l}^{-}\right) ^2}{Z_{\rho _l}^{}}\right) , \end{aligned}$$

(22)

$$\begin{aligned}&=\sum _l Z_{\rho _l}^{} \left( 1{-}\left( \frac{Z_{\rho _l}^{+}}{Z_{\rho _l}^{}}\right) ^2{-}\left( 1{-}\frac{Z_{\rho _l}^{+}}{Z_{\rho _l}^{}}\right) ^2\right) =\sum _l Z_{\rho _l}^{} \underbrace{2 \frac{Z_{\rho _l}^{+}}{Z_{\rho _l}^{}}\left( 1{-}\frac{Z_{\rho _l}^{+}}{Z_{\rho _l}^{}}\right) }_{\varepsilon _{\mathop {{\scriptscriptstyle {\text {gini}}}}\limits _{\rho _{l}}}}. \end{aligned}$$

(23)

We now write out the exponential error and show its connection to Gini error.

(24)

(25)

We aim at showing that , which means that if one removes from a leaf the examples that are not in the m-subset but keeps the tree parameters fixed then the mass times error product must decrease or stay unchanged. We remind that $\rho _l=u_l\cup \bar{u_l}$ and $n>m$, see back to definitions (14). Let us observe the square of using a representation from (25) for the leaf error.

(26)

Note the similarity of the last expression to a Gini representation (22).

We shall now expand (26) taking advantage of the following lemma (for straightforward algebraic proof see [1]) true for either class label $y\in \{-1,+1\}$:

$$\begin{aligned} -\frac{\left( Z_{\rho _l}^{y}\right) ^2}{Z_{\rho _l}^{}} = -\frac{\left( Z_{u_l}^{y}+Z_{\bar{u_l}}^{y}\right) ^2}{Z_{u_l}^{y}+Z_{\bar{u_l}}^{y}} \geqslant -\frac{\left( Z_{u_l}^{y}\right) ^2}{Z_{u_l}^{}}-\frac{\left( Z_{\bar{u_l}}^{y}\right) ^2}{Z_{\bar{u_l}}^{}}. \end{aligned}$$

(27)

Therefore, we have:

(28)

The equality pass comes from grouping odd and even terms using representations (22). Hence, finally: . $\quad \Box$