Convergence Rate of Inertial Proximal Algorithms with General Extrapolation and Proximal Coefficients

Abstract

In a Hilbert space setting ${\mathcal{H}}$, in order to minimize by fast methods a general convex lower semicontinuous and proper function ${\Phi }: {\mathcal{H}} \rightarrow \mathbb {R} \cup \{+\infty \}$, we analyze the convergence rate of the inertial proximal algorithms. These algorithms involve both extrapolation coefficients (including Nesterov acceleration method) and proximal coefficients in a general form. They can be interpreted as the discrete time version of inertial continuous gradient systems with general damping and time scale coefficients. Based on the proper setting of these parameters, we show the fast convergence of values and the convergence of iterates. In doing so, we provide an overview of this class of algorithms. Our study complements the previous Attouch–Cabot paper (SIOPT, 2018) by introducing into the algorithm time scaling aspects, and sheds new light on the Güler seminal papers on the convergence rate of the accelerated proximal methods for convex optimization.

Appendix: Some Auxiliary Results

The following lemmas are used throughout the paper. To establish the weak convergence of the iterates of (IP)$_{\alpha _{k}, \beta _{k}}$, we apply Opial’s Lemma [30], that we recall in its discrete form.

Lemma 4

Let S be a nonempty subset of $\mathcal H$, and (x_k) a sequence in ${\mathcal{H}}$. Assume that

• (i) every sequential weak cluster point of (x_k) as $k\to +\infty$, belongs to S;

• (ii) for every z ∈ S, $\lim _{k\to +\infty }\|x_{k}-z\|$ exists.

Then (x_k) converges weakly as $k \to +\infty$ to a point in S.

Owing to the next lemma, we are able to estimate the rate of convergence of a sequence (ε_k) supposed to be non-increasing and summable with respect to weight coefficients, see [5, Lemma 21] for the proof.

Lemma 5

Let (τ_k) be a nonnegative sequence such that ${\sum }_{k=1}^{+\infty } \tau _{k}=+\infty$. Assume that (ε_k) is a non-negative and non-increasing sequence satisfying ${\sum }_{k=1}^{+\infty } \tau _{k} \varepsilon _{k}<+\infty$. Then we have

$$ \varepsilon_{k} = o\left( \frac{1}{{\sum}_{i=1}^{k} \tau_{i}}\right) \quad \text{ as }~ k\to +\infty. $$

The following result shows the summability of a sequence (a_k) satisfying a suitable inequality.

Lemma 6

Given a non-negative sequence (α_k) satisfying (K₀), let (t_k) be the sequence defined by $t_{k}=1+{\sum }_{i=k}^{+\infty }{\prod }_{j=k}^{i}\alpha _{j}$. Let (a_k) and (ω_k) be two nonnegative sequences such that

$$ a_{k+1} \leq \alpha_{k}a_{k}+\omega_{k}, $$

(51)

for all k ≥ 0. If ${\sum }_{k=0}^{+\infty }t_{k+1}\omega _{k}<+\infty$, then ${\sum }_{k=0}^{+\infty }a_{k}<+\infty$.

Proof

By Lemma 1, we have t_k+ 1α_k = t_k − 1. Multiplying inequality (51) by t_k+ 1 gives

$$ t_{k+1}a_{k+1}\leq (t_{k}-1)a_{k}+t_{k+1}\omega_{k}, $$

or equivalently a_k ≤ (t_ka_k − t_k+ 1a_k+ 1) + t_k+ 1ω_k. By summing from k = 0 to n, we obtain

$$ \sum\limits_{k=0}^{n}a_{k} \leq t_{0} a_{0} - t_{n+1}a_{n+1} + \sum\limits_{k=0}^{n}t_{k+1}\omega_{k} \leq t_{0}a_{0} + \sum\limits_{k=0}^{+\infty}t_{k+1}\omega_{k} < +\infty. $$

The conclusion follows by letting n tend to $+\infty$. □

Lemma 7

[8, Lemma 5.14] Let (a_k) be a sequence of nonnegative numbers such that, for all $k\in \mathbb {N}$, ${a_{k}^{2}} \leq c^{2} + {\sum }_{j=1}^{k} b_{j} a_{j}$, where (b_j) is a summable sequence of nonnegative numbers, and c ≥ 0. Then, for all $k\in \mathbb {N}$, $a_{k} \leq c + {\sum }_{j=1}^{\infty } b_{j}$.