Abstract
In this paper, we propose a Levenberg–Marquardt method based on probabilistic models for nonlinear equations for which the Jacobian cannot be computed accurately or the computation is very expensive. We introduce the definition of the first-order accurate probabilistic Jacobian model, and show how to construct such a model with sample points generated by standard Gaussian distribution. Under certain conditions, we prove that the proposed method converges to a first order stationary point with probability one. Numerical results show the efficiency of the method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this work are available from the authors upon request.
References
Banderia, A.S., Scheinberg, K., Vicente, L.N.: Convergence of trust-region methods based on probabilistic models. SIAM J. Optim. 24(3), 1238–1264 (2014)
Bergou, E., Gratton, S., Vicente, L.N.: Levenberg–Marquardt methods based on probabilistic gradient models and inexact subproblem solution, with application to data assimilation. SIAM/ASA J. Uncertain. Quantif. 4, 924–951 (2016)
Billups, S.C., Larson, J., Graf, P.: Derivative-free optimization of expensive functions with computational error using weighted regression. SIAM J. Optim. 23, 27–53 (2013)
Chen, Z., Dongarra, J.J.: Condition numbers of Gaussian random matrixes. SIAM J. Matrix Anal. Appl. 27, 603–620 (2005)
Conn, A.. R., Scheinberg, K., Toint, Ph.. L.: Approximation Theory and Optimization. In: Powell, M.. J.. D., Buhmann, M.. D., Iserles, A. (eds.) On the convergence of derivative-free methods for unconstrained optimization, pp. 83–108. Cambridge University Press, Cambridge (1997)
Conn, A.R., Scheinberg, K., Vicente, L.N.: Geometry of interpolation sets in derivative free optimization. Math. Program. 111, 141–172 (2008)
Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2009)
Fan, J.Y.: The modified Levenberg–Marquardt method for nonlinear equations with cubic convergence. Math. Comp. 81, 447–466 (2012)
Fan, J.Y., Huang, J.C., Pan, J.Y.: An adaptive multi-step Levenberg–Marquardt method. J. Sci. Comput. 78, 531–548 (2019)
Fan, J.Y., Yuan, Y.X.: On the quadratic convergence of the Levenberg-Marquardt method without nonsingularity assumption. Computing 74, 23–39 (2005)
Ghadimi, S., Lan, G.: Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim. 23, 2341–2368 (2013)
Levenberg, K.: A method for the solution of certain nonlinear problems in least squares. Quardt. Appl. Math. 2, 164–166 (1994)
Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear inequalities. SIAM J. Appl. Math. 11, 431–441 (1963)
Matyas, J.: Random optimization. Autom. Remote Control 26, 246–253 (1965)
Moré, J.J., Garbow, B.S., Hillstrom, K.H.: Testing unconstrained optimization software. ACM Trans. Math. Softw. 7, 17–41 (1981)
Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)
Nesterov, Y.: Random gradient-free minimization of convex functions. Technical Report 2011/1, J. Optim 22, 341–362 (2012)
Polyak. B. T.: Introduction to Optimization, Optimization Software, (1987)
Powell, M.J.D.: Convergence properties of a class of minimization algorithms. In: Mangasarian, O.L., Meyer, R.R., Robinson, S.M. (eds.) Nonlinear Programming 2, pp. 1–27. Academic Press, New York (1975)
Powell, M.. J.. D.: On trust region methods for unconstrained minimization without derivatives. Math. Program. Ser. B 97, 605–623 (2003)
Torczon, V.: On the convergence of pattern search algorithm. SIAM J. Optim. 7, 1–25 (1997)
Vicente, L.N.: Worst case complexity of direct search. EURO J. Comput. Optim. 1, 143–153 (2013)
Yamashita, N., Fukushima, M.: On the rate of convergence of the Levenberg–Marquardt method. Computing 15, 237–249 (2001)
Zhao, R.X., Fan, J.Y.: On a new updating rule of the Levenberg–Marquardt parameter. J. Sci. Comput. 74, 1146–1162 (2018)
Acknowledgements
The authors are grateful to the referees for the valuable comments, which have greatly improved the quality of the paper. They are also indebted to the anonymous referee for the suggestion of constructing the first-order accurate Jacobian models by finite difference method.
Funding
The first author is supported by the Science and Technology Commission of Shanghai Municipality grant 22YF1412400, and the second author is supported by the National Natural Science Foundation of China grant 11971309 and Shanghai Municipal Science and Technology Key Project 22JC1401500.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declared that they have no conflicts of interest to this work.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhao, R., Fan, J. Levenberg–Marquardt method based on probabilistic Jacobian models for nonlinear equations. Comput Optim Appl 83, 381–401 (2022). https://doi.org/10.1007/s10589-022-00393-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-022-00393-9
Keywords
- Levenberg–Marquardt method
- Probabilistic Jacobian models
- Derivative-free optimization
- Global convergence