Large-scale quasi-Newton trust-region methods with low-dimensional linear equality constraints

Abstract

We propose two limited-memory BFGS (L-BFGS) trust-region methods for large-scale optimization with linear equality constraints. The methods are intended for problems where the number of equality constraints is small. By exploiting the structure of the quasi-Newton compact representation, both proposed methods solve the trust-region subproblems nearly exactly, even for large problems. We derive theoretical global convergence results of the proposed algorithms, and compare their numerical effectiveness and performance on a variety of large-scale problems.

Appendix A

Notation

Section 2: Background

${\mathbf {s}}_{k-1}={\mathbf {x}}_{k} - {\mathbf {x}}_{k-1} \qquad \qquad \qquad \qquad \quad {\mathbf {S}}_k =\displaystyle [ {\mathbf {s}}_{k-l} \,\, \cdots \,\, {\mathbf {s}}_{k-1}]$

${\mathbf {y}}_{k-1}=\nabla f({\mathbf {x}}_{k}) - \nabla f({\mathbf {x}}_{k-1}) \qquad \quad {\mathbf {Y}}_k = \displaystyle \left[ {\mathbf {y}}_{k-l} \,\, \cdots \,\, {\mathbf {y}}_{k-1}\right]$

${\mathbf {S}}_k^T {\mathbf {Y}}_k={\mathbf {L}}_k + {\mathbf {T}}_k \qquad \qquad \qquad \qquad \quad {\mathbf {D}}_k=\text {diag}({\mathbf {S}}_k^T{\mathbf {Y}}_k)$

${\mathbf {B}}^{{(k)}}_0={\gamma _{k}} {\mathbf {I}}_n \qquad \qquad \qquad \qquad \qquad \qquad {\mathbf {H}}_k=\mathbf {B}^{-1}_k$

$\gamma _{k}={\mathbf {y}}_{k-1}^T {\mathbf {y}}_{k-1} / {\mathbf {y}}_{k-1}^T {\mathbf {s}}_{k-1} \qquad \qquad \,\, \delta _{k} = {1/\gamma _k}$

${\mathbf {B}}_k=\gamma _k {\mathbf {I}}_n + \widehat{\varvec{\Psi }}_k \widehat{\varvec{\Xi }}_k \widehat{\varvec{\Psi }}_k^T \qquad \qquad \qquad \widehat{\varvec{\Psi }}_k = [ {\mathbf {S}}_k \ \ {\mathbf {Y}}_k]$

${\mathbf {H}}_k=\delta _k {\mathbf {I}}_n + \widehat{\varvec{\Psi }}_k \widehat{{\mathbf {M}}}_k \widehat{\varvec{\Psi }}_k^T$

$\widehat{\varvec{\Xi }}_k = \displaystyle \gamma _k\left[ \begin{array}{cc} - {\mathbf {S}}_k^T {\mathbf {S}}_k &{} - {\mathbf {L}}_k \\ - {\mathbf {L}}_k^T &{} \ \ \gamma _k {\mathbf {D}}_k \end{array}\right] ^{-1}$

$\widehat{{\mathbf {M}}}_k = -(\gamma _k^{2} \widehat{\varvec{\Xi }}_k^{-1} + \gamma _k\widehat{\varvec{\Psi }}_k^T \widehat{\varvec{\Psi }}_k)^{-1}$

Section 3: Trust-Region Subproblem Solution without an Inequality Constraint

${\mathbf {K}}= \displaystyle \left[ \begin{array}{c c} {\mathbf {B}}_k &{} {\mathbf {A}}^T \\ {\mathbf {A}} &{} {\mathbf {0}} \end{array} \right]$                   $\begin{array}{l} \varvec{\Omega }_k = ( {\mathbf {A}} {\mathbf {B}}_k^{-1} {\mathbf {A}}^T )^{-1} \\ \varvec{\Psi }_k =[ {\mathbf {A}}^T \ \ \ \widehat{\varvec{\Psi }}_k ]\end{array}$
$\mathbf {K}^{-1} = \displaystyle \left[ \begin{array}{c c}{\mathbf {B}}_k^{-1} \!- {\mathbf {B}}_k^{-1}{\mathbf {A}}^T \varvec{\Omega }_k {\mathbf {A}} {\mathbf {B}}_k^{-1} \ \ &{} {\mathbf {B}}_k^{-1}{\mathbf {A}}^T \varvec{\Omega }_k \\ ({\mathbf {B}}_k^{-1}{\mathbf {A}}^T \varvec{\Omega }_k)^T \ \ &{} -\varvec{\Omega }_k \\ \end{array} \right]$
${\mathbf {V}}_k = {\mathbf {B}}_k^{-1} \!-\! {\mathbf {B}}_k^{-1}{\mathbf {A}}^T \varvec{\Omega }_k {\mathbf {A}} {\mathbf {B}}_k^{-1}$
${\mathbf {V}}_k = \delta _k {\mathbf {I}}_n + \varvec{\Psi }_k {\mathbf {M}}_k \varvec{\Psi }_k^T$
${\mathbf {W}}_k = {\mathbf {B}}_k^{-1}{\mathbf {A}}^T \varvec{\Omega }_k$
${\mathbf {M}}_k = \displaystyle \left[ \begin{array}{c c} - \delta _k^2 \varvec{\Omega }_k &{} - \delta _k\varvec{\Omega }_k {\mathbf {C}}_k\\ - \delta _k {\mathbf {C}}_k^T \varvec{\Omega }_k &{} \ \widehat{{\mathbf {M}}}_k \!-\! {\mathbf {C}}_k^T\varvec{\Omega }_k{\mathbf {C}}_k \end{array} \right]$
${\mathbf {C}}_k = {\mathbf {A}}\widehat{\varvec{\Psi }}_k\widehat{{\mathbf {M}}}_k$

Section 4: Trust-Region Subproblem Solution with an $\ell _2$-Norm Inequality Constraint

${\mathbf {H}}_k(\sigma ) = ({\mathbf {B}}_k + \sigma {\mathbf {I}})^{-1} \qquad \qquad \qquad \qquad \quad \,\,\, {\mathbf {H}}_k = {\mathbf {H}}_k(0)$

$\varvec{\Phi }_k(\sigma ) = {\mathbf {I}}_n - {\mathbf {A}}^T\varvec{\Omega }_k(\sigma ) {\mathbf {A}}{\mathbf {H}}_k(\sigma ) \qquad \qquad \varvec{\Phi }_k = \varvec{\Phi }_k(0)$

${\mathbf {H}}_k(\sigma ) = \frac{1}{\gamma _k + \sigma }{\mathbf {I}}_n + \widehat{\varvec{\Psi }}_k\widehat{{\mathbf {M}}}_k(\sigma )\widehat{\varvec{\Psi }}_k^T$

$\varvec{\Omega }_k(\sigma ) = ({\mathbf {A}}{\mathbf {H}}_k(\sigma ){\mathbf {A}}^T)^{-1}$

$\widehat{{\mathbf {M}}}_k(\sigma ) = -\big ((\gamma _k + \sigma )^2 \widehat{\varvec{\Xi }}_k^{-1} + (\gamma _k + \sigma )\widehat{\varvec{\Psi }}_k^T\widehat{\varvec{\Psi }}_k \big )^{-1}$

${\mathbf {V}}_k(\sigma ) = {\mathbf {H}}_k(\sigma ) - {\mathbf {H}}_k(\sigma ){\mathbf {A}}^T \varvec{\Omega }_k(\sigma ){\mathbf {A}} {\mathbf {H}}_k(\sigma )$

${\mathbf {V}}_k(\sigma ) = {\mathbf {H}}_k(\sigma ) \varvec{\Phi }_k(\sigma )$

${\mathbf {s}}(\sigma ) = - {\mathbf {H}}_k(\sigma ) \varvec{\Phi }_k(\sigma ) {\mathbf {g}}_k$

${\mathbf {s}}'(\sigma ) = - {\mathbf {H}}_k(\sigma ) \varvec{\Phi }_k(\sigma ) {\mathbf {s}}(\sigma )$

Section 5: Trust-Region Subproblem Solution with a Shape-Changing Norm Inequality Constraint

${\mathbf {U}}_k = -\varvec{\Psi }_k{\mathbf {M}}_k \varvec{\Psi }_k^T$

${\mathbf {A}}^T = \mathbf {Q}_{1} \mathbf {R}_{1}\qquad \qquad \qquad \qquad \qquad \qquad \,\, \mathbf {Q}_{1} \mathbf {Q}_{1}^T = {\mathbf {A}}^T ({\mathbf {A}} {\mathbf {A}}^T)^{-1} {\mathbf {A}}$

${\mathbf {P}} = {\mathbf {I}}_n - {\mathbf {A}}^T ({\mathbf {A}} {\mathbf {A}}^T)^{-1} {\mathbf {A}} \qquad \qquad \qquad {\mathbf {P}}\widehat{\varvec{\Psi }}_k = \widehat{{\mathbf {Q}}}_2\widehat{{\mathbf {R}}}_2$

$\widehat{{\mathbf {V}}}_2\widehat{\varvec{\Lambda }}_k \widehat{{\mathbf {V}}}^T_2 = \widehat{{\mathbf {R}}}_2 (\widehat{{\mathbf {M}}}_k-{\mathbf {C}}_k^T\varvec{\Omega }_k{\mathbf {C}}_k) \widehat{{\mathbf {R}}}^T_2$

$\mathbf {Q}_{2} = \widehat{{\mathbf {Q}}}_2 \widehat{{\mathbf {V}}}_2$

${\mathbf {Q}} = \left[ \mathbf {Q}_{1} \, \mathbf {Q}_{2} \, \mathbf {Q}_{3} \right]$

$\mathbf {Q}_{\parallel } = \left[ \mathbf {Q}_{1} \, \mathbf {Q}_{2} \right] \qquad \qquad \qquad \qquad \qquad \quad \mathbf {Q}_{\perp } = \mathbf {Q}_{3}$

${\mathbf {z}} = \left[ \begin{array}{c} \mathbf {z}_{1} \\ \mathbf {z}_{2} \\ \mathbf {z}_{3} \end{array} \right] \qquad \qquad \qquad \qquad \qquad \qquad \quad {\mathbf {s}} = {\mathbf {Q}} {\mathbf {z}}$

$\mathbf {z}_{\parallel } = \mathbf {z}_{2} = \mathbf {Q}_{2}^T {\mathbf {s}} \qquad \qquad \qquad \qquad \qquad \quad \mathbf {z}_{\perp } = \mathbf {z}_{3} = \mathbf {Q}_{3}^T {\mathbf {s}}$

$\mathbf {g}_{\parallel } = \mathbf {Q}_{2}^T {\mathbf {g}}_k \quad \qquad \qquad \qquad \qquad \qquad \qquad \mathbf {g}_{\perp } = \mathbf {Q}_{\perp }^T {\mathbf {g}}_k$

${\mathbf {V}}_k = {\mathbf {Q}} \varvec{\Lambda } {\mathbf {Q}}^T = \left[ \mathbf {Q}_{1} \, \mathbf {Q}_{2} \, \mathbf {Q}_{3} \right] \left[ \begin{array}{c c c} {\mathbf {0}} &{} \\ &{} \delta _k {\mathbf {I}} - \widehat{\varvec{\Lambda }}_k&{} \\ &{} &{} \delta _k {\mathbf {I}} \end{array} \right] \left[ \begin{array}{c} \mathbf {Q}_{1}^T \\ \mathbf {Q}_{2}^T \\ \mathbf {Q}_{3}^T \\ \end{array} \right]$