Optimizing the Efficiency of First-Order Methods for Decreasing the Gradient of Smooth Convex Functions

Abstract

This paper optimizes the step coefficients of first-order methods for smooth convex minimization in terms of the worst-case convergence bound (i.e., efficiency) of the decrease in the gradient norm. This work is based on the performance estimation problem approach. The worst-case gradient bound of the resulting method is optimal up to a constant for large-dimensional smooth convex minimization problems, under the initial bounded condition on the cost function value. This paper then illustrates that the proposed method has a computationally efficient form that is similar to the optimized gradient method.

Appendix: Proof of Eqs. (25) and (26)

This proof shows the properties (25) and (26) of the step coefficients $\{\tilde{h} _{i,j}\}$ (22).

We first show (25). We can easily derive

$$\begin{aligned} \tilde{h} _{i,i-2} = \frac{(\tilde{\theta } _{i-1}-1)(2\tilde{\theta } _i-1)}{\tilde{\theta } _{i-2}\tilde{\theta } _{i-1}} = \frac{\tilde{\theta } _i^2(2\tilde{\theta } _i-1)}{\tilde{\theta } _{i-2}\tilde{\theta } _{i-1}^2} \end{aligned}$$

for $i=2,\ldots ,N$ using (27). Again using the definition of (22) and (27), we have

$$\begin{aligned} \tilde{h} _{i,j}= & {} \frac{\tilde{\theta } _{j+1}-1}{\tilde{\theta } _j}\tilde{h} _{i,j+1} = \cdots = \left( \prod _{l=j+1}^{i-2}\frac{\tilde{\theta } _l-1}{\tilde{\theta } _{l-1}}\right) \tilde{h} _{i,i-2} = \left( \prod _{l=j+1}^{i-1}\frac{\tilde{\theta } _l-1}{\tilde{\theta } _{l-1}}\right) \frac{2\tilde{\theta } _i-1}{\tilde{\theta } _{i-1}} \\= & {} \frac{1}{\tilde{\theta } _j}\frac{1}{\tilde{\theta } _{j+1}} \frac{\tilde{\theta } _{j+1}-1}{\tilde{\theta } _{j+2}} \cdots \frac{\tilde{\theta } _{i-3}-1}{\tilde{\theta } _{i-2}} (\tilde{\theta } _{i-2}-1)(\tilde{\theta } _{i-1}-1) \frac{2\tilde{\theta } _i-1}{\tilde{\theta } _{i-1}} \\= & {} \frac{1}{\tilde{\theta } _j}\frac{1}{\tilde{\theta } _{j+1}} \frac{\tilde{\theta } _{j+2}}{\tilde{\theta } _{j+1}} \cdots \frac{\tilde{\theta } _{i-2}}{\tilde{\theta } _{i-3}} (\tilde{\theta } _{i-2}-1)(\tilde{\theta } _{i-1}-1) \frac{2\tilde{\theta } _i-1}{\tilde{\theta } _{i-1}} \\= & {} \frac{\tilde{\theta } _{i-2}(\tilde{\theta } _{i-2}-1)(\tilde{\theta } _{i-1}-1)(2\tilde{\theta } _i-1)}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2\tilde{\theta } _{i-1}} = \frac{\tilde{\theta } _i^2(2\tilde{\theta } _i-1)}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2}, \end{aligned}$$

for $i=2,\ldots ,N,\;j=0,\ldots ,i-3$, which concludes the proof of (25).

We next prove the first two lines of (26) using the induction. For $N=1$, we have $\tilde{\theta } _1 = 1$ and

$$\begin{aligned} \tilde{h} _{1,0} = 1 + \frac{2\tilde{\theta } _1-1}{\tilde{\theta } _0} = 1 + \frac{\tilde{\theta } _1^2}{\tilde{\theta } _0} = 1 + \frac{\frac{1}{2}(\tilde{\theta } _0^2 - \tilde{\theta } _0)}{\tilde{\theta } _0} = \frac{1}{2}(\tilde{\theta } _0+1) , \end{aligned}$$

where the third equality uses (27). For $N>1$, we have

$$\begin{aligned} \tilde{h} _{N,N-1} = 1 + \frac{2\tilde{\theta } _N-1}{\tilde{\theta } _{N-1}} = 1 + \frac{\tilde{\theta } _N^2}{\tilde{\theta } _{N-1}} = 1 + \frac{\tilde{\theta } _{N-1}^2 - \tilde{\theta } _{N-1}}{\tilde{\theta } _{N-1}} = \tilde{\theta } _{N-1} , \end{aligned}$$

where the third equality uses (27). Assuming $\sum _{l=j+1}^N\tilde{h} _{l,j} = \tilde{\theta } _j$ for $j=n,\ldots ,N-1$ and $n\ge 1$, we get

$$\begin{aligned} \sum _{l=n}^N\tilde{h} _{l,n-1}&= 1 + \frac{2\tilde{\theta } _n-1}{\tilde{\theta } _{n-1}} + \frac{\tilde{\theta } _n-1}{\tilde{\theta } _{n-1}}(\tilde{h} _{n+1,n}-1) + \frac{\tilde{\theta } _n-1}{\tilde{\theta } _{n-1}}\sum _{l=n+2}^N\tilde{h} _{l,n} \\&= 1 + \frac{\tilde{\theta } _n}{\tilde{\theta } _{n-1}} + \frac{\tilde{\theta } _n-1}{\tilde{\theta } _{n-1}}\sum _{l=n+1}^N\tilde{h} _{l,n} = \frac{\tilde{\theta } _{n-1} + \tilde{\theta } _n + (\tilde{\theta } _n-1)\tilde{\theta } _n}{\tilde{\theta } _{n-1}} = \frac{\tilde{\theta } _{n-1} + \tilde{\theta } _n^2}{\tilde{\theta } _{n-1}} \\&= {\left\{ \begin{array}{ll} \frac{1}{2}(\tilde{\theta } _0 + 1), &{} n = 0, \\ \tilde{\theta } _n, &{} n=1,\ldots ,N-1, \end{array}\right. } \end{aligned}$$

where the last equality uses (27), which concludes the proof of the first two lines of (26).

We finally prove the last line of (26) using the induction. For $i\ge 1$, we have

$$\begin{aligned} \sum _{l=i+1}^N\tilde{h} _{l,i-1} = \sum _{l=i}^N\tilde{h} _{l,i-1} - \tilde{h} _{i,i-1} = \tilde{\theta } _{i-1} - \left( 1+\frac{2\tilde{\theta } _i-1}{\tilde{\theta } _{i-1}}\right) = \frac{(\tilde{\theta } _i-1)^2}{\tilde{\theta } _{i-1}} = \frac{\tilde{\theta } _{i+1}^4}{\tilde{\theta } _{i-1}\tilde{\theta } _i^2} , \end{aligned}$$

where the third and fourth equalities use (27). Then, assuming $\sum _{l=i+1}^N\tilde{h} _{l,j}=\frac{\tilde{\theta } _i^4}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2}$ for $i=n,\ldots ,N-1$, $j=0,\ldots ,i-1$ with $n\ge 1$, we get:

$$\begin{aligned} \sum _{l=n}^N\tilde{h} _{l,j}&= \sum _{l=n+1}^N\tilde{h} _{l,j} + \tilde{h} _{n,j} = \frac{\tilde{\theta } _{n+1}^4}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2} + \frac{\tilde{\theta } _n^2(2\tilde{\theta } _n-1)}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2} = \frac{\tilde{\theta } _n^2(\tilde{\theta } _n-1)^2 + \tilde{\theta } _n^2(2\tilde{\theta } _n-1)}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2}\\&= \frac{\tilde{\theta } _n^4}{\tilde{\theta } _j\tilde{\theta } _{j+1}^2} , \end{aligned}$$

where the second and third equalities use (25), which concludes the proof. $\square$