Abstract
In this article, we propose a method for solving unconstrained optimization problems with convex and Lipschitz continuous objective functions. By making use of the Moreau envelopes of the functions occurring in the objective, we smooth them to convex and differentiable functions with Lipschitz continuous gradients using both variable and constant smoothing parameters. The resulting problem is solved via an accelerated first-order method and this allows us to recover approximately the optimal solutions to the initial optimization problem with a rate of convergence of order \(\mathcal {O}\left( \tfrac{\ln k}{k}\right) \) for variable smoothing and of order \(\mathcal {O}\left( \tfrac{1}{k}\right) \) for constant smoothing. Some numerical experiments employing the variable smoothing method in image processing and in supervised learning classification are also presented.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bauschke HH, Combettes PL (2011) Convex analysis and monotone operator theory in Hilbert spaces. In: CMS books in mathematics., Springer, New York
Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202
Beck A, Teboulle M (2012) Smoothing and first order methods: a unified framework. SIAM J Optim 22(2):557–580
Bertsekas D (1996) Constrained optimization and Lagrange Multiplier Methods. Athena Scientific, Belmont
Borwein JM, Vanderwerff JD (2010) Convex functions: constructions. Characterizations and counterexamples. Cambridge University Press, Cambridge
Boţ RI (2010) Conjugate duality in convex optimization. In: Lecture notes in economics and mathematical systems, vol 637. Springer, Berlin
Boţ RI, Hendrich C (2012) On the acceleration of the double smoothing technique for unconstrained convex optimization problems. Optimization. doi:10.1080/02331934.2012.745530
Boţ RI, Hendrich C (2013) A double smoothing technique for solving unconstrained nondifferentiable convex optimization problems. Comput Optim Appl 54(2):239–262
Boţ RI, Hendrich C (2013) A Douglas–Rachford type primal–dual method for solving inclusions with mixtures of composite and parallel-sum type monotone operators. SIAM J Optim 23(4):2541–2565
Boţ RI, Heinrich A, Wanka G (2014) Employing different loss functions for the classification of images via supervised learning. Central Eur J Math 12(2):381–394
Boţ RI, Lorenz N (2011) Optimization problems in statistical learning: duality and optimality conditions. Eur J Oper Res 213(2):395–404
Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2010) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122
Briceño-Arias LM, Combettes PL (2011) A monotone + skew splitting model for composite monotone inclusions in duality. SIAM J Optim 21(4):1230–1250
Burachik RS, Jeyakumar V (2005) A new geometric condition for Fenchel’s duality in infinite dimensional spaces. Math Progr 104(2–3):229–233
Chambolle A, Pock T (2011) A first-order primal–dual algorithm for convex problems with applications to imaging. J Math Imaging Vis 40(1):120–145
Chantas G, Galatsanos N, Likas A, Saunders M (2008) Variational Bayesian image restoration based on a product of t-distributions image prior. IEEE Trans Image Process 17(10):1795–1805
Combettes PL, Pesquet JC (2011) Proximal splitting methods in signal processing. In: Bauschke HH et al Fixed-point algorithms for inverse problems in science and engineering. Springer Ser. Optim. Appl. vol 49. Springer, New York pp 185–212
Combettes PL, Pesquet J-C (2012) Primal–dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set Valued Var Anal 20(2):307–330
Combettes PL, Wajs VR (2005) Signal recovery by proximal forward–backward splitting. Multiscale Model Simul 4(4):1168–1200
Devolder O, Glineur F, Nesterov Y (2012) Double smoothing technique for large-scale linearly constrained convex optimization. SIAM J Optim 22(2):702–727
Lal TN, Chapelle O, Schölkopf B (2006) Combining a filter method with SVMs. In: Studies in fuzziness and soft computing, vol 207. Springer, Berlin, pp 439–445
Li G, Ng KF (2008) On extension of Fenchel duality and its application. SIAM J Optim 19(3):1489–1509
Moreau JJ (1965) Proximité et dualitè dans un espace hilbertien Bull. Soc Math Fr 93:273–299
Nesterov Y (1983) A method for unconstrained convex minimization problem with the rate of convergence \({\cal O}(1/k^2)\). Doklady AN SSSR (translated as Soviet Math. Docl.) 269:543–547
Nesterov Y (2004) Introductory lectures on convex optimization: a basic course. Kluwer Academic Publishers, Dordrecht
Nesterov Y (2005) Excessive gap technique in nonsmooth convex optimization. SIAM J Optim 16(1):235–249
Nesterov Y (2005) Smooth minimization of non-smooth functions. Math Progr 103(1):127–152
Nesterov Y (2005) Smoothing technique and its applications in semidefinite optimization. Math Progr 110(2):245–259
Orabona F, Argyriou A, Srebro N (2012) PRISMA: PRoximal Iterative SMoothing Algorithm. arXiv:1206.2372 [math.OC]
Shawe-Taylor J, Christianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, Cambridge
Simons S (2008) From Hahn–Banach to monotonicity. Springer, Berlin
Vũ BC (2013) A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv Comput Math 38(3):667–681
Acknowledgments
The authors are thankful to two anonymous reviewers for hints and remarks which improved the quality of the paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
R. I. Boţ was partially supported by DFG (German Research Foundation), project BO 2516/4-1. C. Hendrich was supported by a Graduate Fellowship of the Free State Saxony, Germany.
Rights and permissions
About this article
Cite this article
Boţ, R.I., Hendrich, C. A variable smoothing algorithm for solving convex optimization problems. TOP 23, 124–150 (2015). https://doi.org/10.1007/s11750-014-0326-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11750-014-0326-z