Abstract
In this paper, we develop an active set identification technique for the \(\ell _0\) regularization optimization. Such a technique has a strong ability to identify the zero components in a neighbourhood of a strict L-stationary point. Based on the identification technique, we propose an active set Barzilar–Borwein algorithm and prove that any limit point of the sequence generated by the algorithm is a strong stationary point. Some preliminary numerical results are provided, showing that the method is promising.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137, 91–129 (2013)
Barzilai, J., Borwein, J.M.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)
Beck, A., Hallak, N.: Proximal mapping for symmetric penalty and sparsity. SIAM J. Optim. 28, 496–527 (2018)
Bi, S.J., Liu, X.L., Pan, S.H.: Exact penalty decomposition method for zero norm minimization based on MPEC formulation. SIAM J. Sci. Comput. 36, 1451–1477 (2014)
Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14, 629–654 (2008)
Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)
Breiman, L.: Better subset regression using the nonnegative garrote. Technometrics 37, 373–384 (1995)
Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 489–509 (2006)
Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14, 877–905 (2008)
Chen, T.Y., Curtis, F.E., Robinson, D.P.: A reduced-space algorithm for minimizing \(\ell _1\)-regularized convex functions. SIAM J. Optim. 3, 1583–1610 (2017)
Cheng, W.Y., Dai, Y.H.: Gradient-based method with active set strategy for \(\ell _1\) optimization. Math. Comput. 87, 1283–1305 (2018)
Dassios, I., Fountoulakis, K., Gondzio, J.: A preconditioner for a primal-dual newton conjugate gradient method for compressed sensing problems. SIAM J. Sci. Comput. 37, 2783–2812 (2015)
Davis, G., Mallat, S., Avellaneda, M.: Adaptive greedy approximations. Constr. Approx. 13, 57–98 (1997)
Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57, 1413–1457 (2004)
Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)
Donoho, D.L.: For most large underdetermined systems of linear equations the minimal \(\ell _1\)-norm solution is also the sparsest solution. Commun. Pure App. Math. 59, 797–829 (2006)
Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96, 1348–1360 (2001)
Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1, 586–597 (2007)
Foucart, S., Lai, M.J.: Sparsest solutions of underdetermined linear systems via \(\ell _q\)(\(0<q<1\)). Appl. Comput. Harmon. Anal. 26, 395–407 (2009)
Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986)
Hale, E.T., Yin, W., Zhang, Y.: Fixed-point continuation for \(\ell _1\) minimization: methodology and convergence. SIAM J. Optim. 19, 1107–1130 (2008)
Jiao, Y.L., Jin, B.J., Lu, X.L.: A primal dual active set with continuation algorithm for the \(\ell _0\)-regularized optimization problem. Appl. Comput. Harmon. Anal. 39, 400–426 (2015)
Liang, J.W., Fadili, J., Peyré, G.: Activity identification and local linear convergence of forward–backward-type methods. SIAM J. Optim. 1, 408–437 (2017)
Liu, Y.L., Pan, S.H., Bi, S.J.: Equivalent Lipschitz surrogates for zero-norm and rank optimization problems. J. Glob. Optim. 72, 679–704 (2018)
Lu, Z., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim. 23, 2448–2478 (2013)
Mallat, S.G., Zhang, Z.: Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 41, 3397–3415 (1993)
Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)
Needell, D., Tropp, J.A.: CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26, 301–321 (2009)
Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In: Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers (Pacific Grove, CA), pp. 40–44. IEEE, Washington, DC (1993)
Santis, M.D., Lucidi, S., Rinaldi, F.: A fast active set block coordinate descent algorithm for \(\ell _1\)-regularized least squares. SIAM. J. Optim. 26, 781–809 (2016)
Soubies, E., Blanc-Féraud, L., Aubert, G.: A unified view of exact continuous penalties for \(\ell _2\)-\(\ell _0\) minimization. SIAM J. Optim. 27, 2034–2060 (2017)
Soussen, C., Idier, J., Brie, D., Duan, J.: From Bernoulli–Gaussian deconvolution to sparse signal restoration. IEEE Trans. Signal Process. 59, 4572–4584 (2011)
Soubies, E., Blanc-Féraud, L., Aubert, G.: A continuous exact \(\ell _0\) penalty (CEL0) for least squares regularized problem. SIAM J. Imaging Sci. 8, 1607–1639 (2015)
Temlyakov, V.N.: Greedy approximation. Acta Numer. 17, 235–409 (2008)
Tseng, P., Yun, S.W.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117, 387–423 (2009)
Wen, Z.W., Yin, W.T., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization, and continuation. SIAM J. Sci. Comput. 32, 1832–1857 (2010)
Wright, J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)
Xu, Z.B., Chang, X.Y., Xu, F.M., Zhang, H.: \(\ell _{\frac{1}{2}}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. 23, 1013–1027 (2012)
Yuan, G.X., Ho, C.H., Lin, C.J.: Recent advances of large-scale linear classification. Proc. IEEE 100, 2584–2603 (2012)
Zhang, T.: Multi-stage convex relaxation for learning with sparse regularization. Adv. Neural Inf. Process. Syst. 22, 1929–1936 (2009)
Zhang, C.H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38, 894–942 (2010)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supported by the Chinese NSF Grant (Nos. 11371154, 11961101, 11761014, 11971106, and 11461015), by the Ministry of Education, Humanities and Social Sciences Project (No. 17JYJAZH011) and by the Natural Science Foundation of Guangdong province (2018A030313229)
Rights and permissions
About this article
Cite this article
Cheng, W., Chen, Z. & Hu, Q. An active set Barzilar–Borwein algorithm for \(l_{0}\) regularized optimization. J Glob Optim 76, 769–791 (2020). https://doi.org/10.1007/s10898-019-00830-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-019-00830-w