Direct search based on probabilistic feasible descent for bound and linearly constrained problems

Abstract

Direct search is a methodology for derivative-free optimization whose iterations are characterized by evaluating the objective function using a set of polling directions. In deterministic direct search applied to smooth objectives, these directions must somehow conform to the geometry of the feasible region, and typically consist of positive generators of approximate tangent cones (which then renders the corresponding methods globally convergent in the linearly constrained case). One knows however from the unconstrained case that randomly generating the polling directions leads to better complexity bounds as well as to gains in numerical efficiency, and it becomes then natural to consider random generation also in the presence of constraints. In this paper, we study a class of direct-search methods based on sufficient decrease for solving smooth linearly constrained problems where the polling directions are randomly generated (in approximate tangent cones). The random polling directions must satisfy probabilistic feasible descent, a concept which reduces to probabilistic descent in the absence of constraints. Such a property is instrumental in establishing almost-sure global convergence and worst-case complexity bounds with overwhelming probability. Numerical results show that the randomization of the polling directions can be beneficial over standard approaches with deterministic guarantees, as it is suggested by the respective worst-case complexity bounds.

Appendices

A Deterministic computation of positive cone generators

Provided a certain linear independence condition is satisfied, the positive generators of the approximate tangent cone can be computed as follows (see [25, Proposition 5.2]).

Proposition A.1

Let $x \in \mathcal {F}$, $\alpha > 0$, and assume that the set of generators of $N(x,\alpha )$ has the form $\left[ \, W_e\ \, -\!W_e\ \; W_i \,\right]$. Let B be a basis for the null space of $W_e^\top$, and suppose that $W_i^\top B = Q^\top$ has full row rank. If R is a right inverse for $Q^\top$ and N is a matrix whose columns form a basis for the null space of $Q^\top$, then the set

$$\begin{aligned} Y = \left[ \; -\!\,BR \quad BN \ \; -\!BN\; \right] \end{aligned}$$

(A.1)

positively generates $T(x,\alpha )$.

Note that the number of vectors in Y is then $n_R+2(n_B-n_R)=2n_B - n_R$, where $n_B$ is the rank of B and $n_R$ is that of Q (equal to number of columns of R). Since $n_B < {\tilde{n}}$, we have that $|Y| \le 2 {\tilde{n}}$.

In the case where $W_i^\top B$ is not full row rank, one could consider all subsets of columns of $W_i$ of largest size that yield full row rank matrices, obtain the corresponding positive generators by Proposition A.1, and then take the union of all these sets [34]. Due to the combinatorial nature of this technique, we adopted a different approach, following the lines of [22, 25] where an algorithm originating from computational geometry called the double description method [16] was applied. We implemented this algorithm to compute a set of extreme rays (or positive generators) for a polyhedral cone of the form $\{ {\tilde{d}} \in \mathbb {R}^{{\tilde{n}}}: B {\tilde{d}} = 0, C {\tilde{d}} \ge 0 \}$, where we assume that ${{\,\mathrm{rank}\,}}(B) < {\tilde{n}}$, and applied it to the approximate tangent cone. In our experiments, the three variants detected degeneracy (and thus invoked the double description method) in less than 2% of the iterations on average.

B Proof of Proposition 7.1

Proof

To simplify the notation, we omit the index k in the proof. By our convention about ${{\,\mathrm{cm}\,}}_T(V, -{\tilde{G}})$ when $P_T[{\tilde{G}}]=0$, we only need to consider the situation where $P_T[{\tilde{G}}]$ is nonzero.

Define the event

$$\begin{aligned} \mathcal {E} = \left\{ \dfrac{\Vert P[{\tilde{G}}]\Vert }{\Vert P_{T}[{\tilde{G}}]\Vert } \ge \dfrac{1}{\sqrt{2}}\right\} \end{aligned}$$

and let $\bar{\mathcal {E}}$ denote its complementary event. We observe that

$$\begin{aligned} \left\{ {{\,\mathrm{cm}\,}}_{T}([U^s\;\; U^c], -{\tilde{G}}) \ge \kappa \right\} \supset&\left\{ {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \mathcal {E}, \end{aligned}$$

(B.1)

$$\begin{aligned} \left\{ {{\,\mathrm{cm}\,}}_{T}([U^s\;\; U^c], -{\tilde{G}}) \ge \kappa \right\} \supset&\left\{ {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \bar{\mathcal {E}} \end{aligned}$$

(B.2)

for each $\kappa$. Indeed, when $\mathcal {E}$ happens, we have

$$\begin{aligned}&{{\,\mathrm{cm}\,}}_{T}([U^s\;\; U^c], -{\tilde{G}}) \\&\quad \ge {{\,\mathrm{cm}\,}}_{T}(U^s, -{\tilde{G}})= \dfrac{\Vert P_S[{\tilde{G}}]\Vert }{\Vert P_{T}[{\tilde{G}}]\Vert }{{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \dfrac{1}{\sqrt{2}}{{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}), \end{aligned}$$

and (B.1) is consequently true; when $\bar{\mathcal {E}}$ happens, then by the orthogonal decomposition

$$\begin{aligned} \Vert P_{T}[{\tilde{G}}]\Vert ^2 = \Vert P_{S}[{\tilde{G}}]\Vert ^2 + \Vert P_{T^c}[{\tilde{G}}]\Vert ^2, \end{aligned}$$

we know that $\Vert P_{T^c}[{\tilde{G}}]\Vert /\Vert P_{T}[{\tilde{G}}]\Vert \ge 1/\sqrt{2}$, and hence

$$\begin{aligned} {{\,\mathrm{cm}\,}}_{T}([U^s\;\; U^c], -{\tilde{G}})\ge & {} {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \\= & {} \dfrac{\Vert P_{T^c}[{\tilde{G}}]\Vert }{\Vert P_{T}[{\tilde{G}}]\Vert }{{\,\mathrm{cm}\,}}_{T^{c}}(U^c, -{\tilde{G}}) \\\ge & {} \dfrac{1}{\sqrt{2}}{{\,\mathrm{cm}\,}}_{T^{c}}(U^c, -{\tilde{G}}), \end{aligned}$$

which gives us (B.2). Since the events on the right-hand sides of (B.1) and (B.2) are mutually exclusive, the two inclusions lead us to

$$\begin{aligned} \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{T}([U^s\;\; U^c], -{\tilde{G}}) \ge \kappa \mathrel {\big |} \sigma \right)\ge & {} \mathbb {P}\left( \left\{ {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \mathcal {E} \mathrel {\big |} \sigma \right) \\&+\,\mathbb {P}\left( \left\{ {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \bar{\mathcal {E}} \mathrel {\big |}\sigma \right) . \end{aligned}$$

Then it suffices to show the existence of $\kappa \in (0,1)$ and $p \in (p_0,1)$ that fulfill simultaneously

$$\begin{aligned} \mathbb {P}\left( \left\{ {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \mathcal {E} \mathrel {\big |} \sigma \right)&\ge p \,{\mathbf {1}}_{\mathcal {E}}, \end{aligned}$$

(B.3)

$$\begin{aligned} \mathbb {P}\left( \left\{ {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \bar{\mathcal {E}} \mathrel {\big |} \sigma \right)&\ge p {\mathbf {1}}_{\bar{\mathcal {E}}}, \end{aligned}$$

(B.4)

because ${\mathbf {1}}_{\mathcal {E}} + {\mathbf {1}}_{\bar{\mathcal {E}}}=1$. If $\mathbb {P}(\mathcal {E}) = 0$, then (B.3) holds trivially regardless of $\kappa$ or p. Similar things can be said about $\bar{\mathcal {E}}$ and (B.4). Therefore, we assume that both $\mathcal {E}$ and $\mathcal {{\bar{E}}}$ are of positive probabilities. Then, noticing that $\mathcal {E}\in \sigma$ ($\mathcal {E}$ only depends on the past iterations), we have

$$\begin{aligned} \mathbb {P}\left( \left\{ {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \mathcal {E} \mathrel {\big |} \sigma \right)&= \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \mathrel {\big |} \sigma \cap \mathcal {E} \right) \mathbb {P}(\mathcal {E}|\sigma ), \\&=\mathbb {P}\left( {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \mathrel {\big |} \sigma \cap \mathcal {E} \right) {\mathbf {1}}_{\mathcal {E}}, \\ \mathbb {P}\left( \left\{ {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \right\} \cap \bar{\mathcal {E}} \mathrel {\big |} \sigma \right)&= \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \mathrel {\big |} \sigma \cap \bar{\mathcal {E}} \right) \mathbb {P}(\bar{\mathcal {E}}|\sigma )\\&= \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \mathrel {\big |} \sigma \cap \bar{\mathcal {E}} \right) {\mathbf {1}}_{\bar{\mathcal {E}}}, \end{aligned}$$

where $\sigma \cap \mathcal {E}$ is the trace $\sigma$-algebra [35] of $\mathcal {E}$ in $\sigma$, namely $\sigma \cap \mathcal {E} = \left\{ \mathcal {A} \cap \mathcal {E} \mathrel : \mathcal {A} \in \sigma \right\}$, and $\sigma \cap \bar{\mathcal {E}}$ is that of $\bar{\mathcal {E}}$. Hence it remains to prove that

$$\begin{aligned} \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{S}(U^s, -{\tilde{G}}) \ge \sqrt{2}\kappa \mathrel {\big |} \sigma \cap \mathcal {E} \right)&\ge p, \end{aligned}$$

(B.5)

$$\begin{aligned} \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}}) \ge \sqrt{2}\kappa \mathrel {\big |} \sigma \cap \bar{\mathcal {E}} \right)&\ge p. \end{aligned}$$

(B.6)

Let us examine ${{\,\mathrm{cm}\,}}_S(U^s, -{\tilde{G}})$ whenever $\mathcal {E}$ happens (which necessarily means that S is nonzero). Since $\mathcal {E}$ depends only on the past iterations, while $U^s$ is essentially a set of $r_s$ (recall (7.1)) i.i.d. directions from the uniform distribution on the unit sphere in $\mathbb {R}^s$ with $s=\dim (S) \le {\tilde{n}}$, one can employ the theory of [19, Appendix B] to justify the existence of $p_s > p_0$ and $\tau > 0$ that are independent of ${\tilde{n}}$ or s (solely depending on $\theta$ and $\gamma$) and satisfy

$$\begin{aligned} \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{S}(U^s,-{\tilde{G}}) \ge \frac{\tau }{\sqrt{{\tilde{n}}}} \mathrel {\Big |} \sigma \, \cap \, \mathcal {E} \right) \ge p_s. \end{aligned}$$

(B.7)

Now consider ${{\,\mathrm{cm}\,}}_{T^c}(U^c, -{\tilde{G}})$ under the occurrence of $\bar{\mathcal {E}}$ (in that case, $T^c$ is nonzero and $V^c$ is nonempty). Let $D^*$ be a direction in $V^c$ that achieves

$$\begin{aligned} \frac{D^{*\top } (-{\tilde{G}})}{\Vert D^*\Vert \Vert P_{T^c}[{\tilde{G}}]\Vert } = {{\,\mathrm{cm}\,}}_{T^c} (V^c, -{\tilde{G}}). \end{aligned}$$

Then by the fact that $U^c$ is a uniform random subset of $V^c$ and $|U^c| = \lceil p_c |V^c| \rceil$, we have

$$\begin{aligned} \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{T^c}( U^c,-{\tilde{G}}) = {{\,\mathrm{cm}\,}}_{T^c} (V^c, -{\tilde{G}}) \mathrel {\big |} \sigma \cap {\bar{\mathcal {E}}} \right) \ge \mathbb {P}\left( D^* \in U^c \mathrel {\big |} \sigma \cap {\bar{\mathcal {E}}} \right) = \frac{|U^c|}{|V^c|} \ge p_c.\nonumber \\ \end{aligned}$$

(B.8)

Let

$$\begin{aligned} \kappa _c = \lambda \left( \big \{ C = {\bar{T}} \cap {\bar{S}}^\perp : \; {\bar{T}} \in {\mathbf {T}} \text { and } {\bar{S}}\ \text { is a subspace of } {\bar{T}} \big \} \right) , \end{aligned}$$

where $\lambda (\cdot )$ is defined as in (3.2) and ${\mathbf {T}}$ denotes all possible occurrences for the approximate tangent cone. Then $\kappa _c > 0$ and ${{\,\mathrm{cm}\,}}_{T^c} (V^c, -{\tilde{G}}) \ge \kappa _c$. Hence (B.8) implies

$$\begin{aligned} \mathbb {P}\left( {{\,\mathrm{cm}\,}}_{T^c}( U^c,-{\tilde{G}}) \ge \kappa _c \mathrel {\big |} \sigma \cap {\bar{\mathcal {E}}}\right) \ge p_c. \end{aligned}$$

(B.9)

Finally, set

$$\begin{aligned} \kappa = \frac{1}{\sqrt{2}}\min \left\{ \frac{\tau }{\sqrt{{\tilde{n}}}}, \; \kappa _c \right\} , \quad p = \min \left\{ p_s,\; p_c \right\} . \end{aligned}$$

Then $\kappa \in (0,1)$, $p \in (p_0, 1)$, and they fulfill (B.5) and (B.6) according to (B.7) and (B.9). Moreover, $\kappa$ depends on the geometry of T, and consequently depends on m, n, and $m_{\mathcal {I}}$, while p depends solely on $\theta$, $\gamma$, and $p_c$. The proof is then completed. $\square$

C List of CUTEst problems

The complete list of our test problems, with the associated dimensions and number of constraints, is provided in Tables 1, 2, 3 and 4.

Table 1 List of the smallest tested CUTEst test problems with bound constraints

Table 2 List of the largest tested CUTEst test problems with bound constraints

Table 3 List of the tested CUTEst test problems with only linear equality constraints and bounds

Table 4 List of the tested CUTEst test problems with at least one linear inequality constraint