Skip to main content
SpringerLink
Log in

An efficient Gauss–Newton algorithm for solving regularized total least squares problems

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

The total least squares (TLS) method is a well-known technique for solving an overdetermined linear system of equations Ax ≈ b, that is appropriate when both the coefficient matrix A and the right-hand side vector b are contaminated by some noise. For ill-posed TLS poblems, regularization techniques are necessary to stabilize the computed solution; otherwise, TLS produces a noise-dominant output. In this paper, we show that the regularized total least squares (RTLS) problem can be reformulated as a nonlinear least squares problem and can be solved by the Gauss–Newton method. Due to the nature of the RTLS problem, we present an appropriate method to choose a good regularization parameter and also a good initial guess. Finally, the efficiency of the proposed method is examined by some test problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Global convergence means convergence to a local minimum from any initial point [22].

References

  1. Adcock, R. J.: Note on the method of least squares. Analyst 4 (6), 183–184 (1877)

    Article  Google Scholar 

  2. Beck, A., Ben-Tal, A.: On the solution of the Tikhonov regularization of the total least squares problem. SIAM J. Optim. 17(1), 98–118 (2006)

    Article  MathSciNet  Google Scholar 

  3. Beck, A., Ben-Tal, A., Teboulle, M.: Finding a global optimal solution for a quadratically constrained fractional quadratic problem with applications to the regularized total least squares. SIAM J. Matrix Anal. Appl. 28(2), 425–445 (2006)

    Article  MathSciNet  Google Scholar 

  4. Björck, Å.: Numerical methods for least squares problems. SIAM (1996)

  5. Björck, Å., Heggernes, P.x, Matstoms, P.: Methods for large scale total least squares problems. SIAM J. Matrix Anal. Appl. 22(2), 413–429 (2000)

    Article  MathSciNet  Google Scholar 

  6. Buccini, A., Park, Y., Reichel, L.: Comparison of a-posteriori parameter choice rules for linear discrete ill-posed problems. J. Comput. Appl. Math. 373, 112138 (2020)

    Article  MathSciNet  Google Scholar 

  7. Calvetti, D., Golub, G. H., Reichel, L.: Estimation of the L-curve via lanczos bidiagonalization. BIT Numer. Math. 39(4), 603–619 (1999)

    Article  MathSciNet  Google Scholar 

  8. Calvetti, D., Reichel, L., Shuibi, A.: L-Curve and curvature bounds for Tikhonov regularization. Numer. Algorithm. 35(2), 301–314 (2004)

    Article  MathSciNet  Google Scholar 

  9. Coello, C. A. C., Lamont, G. B., Van Veldhuizen, D. A., et al.: Evolutionary Algorithms for Solving Multi-Objective Problems, vol. 5. Springer (2007)

  10. Cortiella, A., Park, K. -C., Doostan, A.: Sparse identification of nonlinear dynamical systems via reweighted ℓ1-regularized least squares. Comput. Methods Appl. Mech. Eng. 376, 113620 (2021)

    Article  MathSciNet  Google Scholar 

  11. Deb, K.: Multi-objective Optimization Using Evolutionary Algorithms, vol. 16. Wiley (2001)

  12. Ehrgott, M.: Multicriteria Optimization, vol. 491. Springer Science & Business Media (2005)

  13. Engl, H. W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems, vol. 375. Springer Science & Business Media (2000)

  14. Fasino, D., Fazzi, A.: A Gauss–Newton iteration for total least squares problems. BIT Numer. Math., 1–19 (2018)

  15. Fenu, C., Reichel, L., Rodriguez, G., Sadok, H.: GCV for Tikhonov regularization by partial SVD. BIT Numer. Math. 57(4), 1019–1039 (2017)

    Article  MathSciNet  Google Scholar 

  16. Fierro, R. D., Golub, G. H., Hansen, P. C., O’Leary, D. P.: Regularization by truncated total least squares. SIAM J. Sci. Comput. 18(4), 1223–1241 (1997)

    Article  MathSciNet  Google Scholar 

  17. Gander, W., Golub, G. H., von Matt, U.: A constrained eigenvalue problem. Linear Algebra Appl. 114, 815–839 (1989)

    Article  MathSciNet  Google Scholar 

  18. Golub, G. H., Hansen, P. C., O’Leary, D. P.: TIkhonov regularization and total least squares. SIAM J. Matrix Anal. Appl. 21(1), 185–194 (1999)

    Article  MathSciNet  Google Scholar 

  19. Golub, G. H., Van Loan, C. F.: An analysis of the total least squares problem. SIAM J. Numer. Anal. 17(6), 883–893 (1980)

    Article  MathSciNet  Google Scholar 

  20. Golub, G. H., Van Loan, C. F.: Matrix Computations. Johns Hopkins University Press, Baltimore (2013)

  21. Guo, H., Renaut, R. A.: A regularized total least squares algorithm. In Total Least Squares and Errors-in-Variables Modeling, pp. 57–66. Springer (2002)

  22. Hansen, P., Pereyra, V., Scherer, G.: Least squares data fitting with applications. Johns Hopkins University Press (2013)

  23. Hansen, P. C.: Rank-deficient and Discrete Ill-posed Problems: Numerical aspects of linear inversion. SIAM (1998)

  24. Hansen, P. C.: Regularization tools version 4.0 for matlab 7.3. Numer. Algorithm. 46(2), 189–194 (2007)

    Article  MathSciNet  Google Scholar 

  25. Hansen, P. C., O’Leary, D. P.: The use of the L-curve in the regularization of discrete ill-posed problems. SIAM J. Sci. Comput. 14(6), 1487–1503 (1993)

    Article  MathSciNet  Google Scholar 

  26. Lampe, J., Voss, H.: On a quadratic eigenproblem occurring in regularized total least squares. Comput. Stat. Data Anal. 52(2), 1090–1102 (2007)

    Article  MathSciNet  Google Scholar 

  27. Lampe, J., Voss, H.: A fast algorithm for solving regularized total least squares problems. Electron. Trans. Numer Anal. 31, 12–24 (2008)

    MathSciNet  MATH  Google Scholar 

  28. Lampe, J., Voss, H.: Solving regularized total least squares problems based on eigenproblems. Taiwan. J. Math. 14(3A), 885–909 (2010)

    Article  MathSciNet  Google Scholar 

  29. Lampe, J., Voss, H.: Efficient determination of the hyperparameter in regularized total least squares problems. Appl. Numer. Math. 62(9), 1229–1241 (2012)

    Article  MathSciNet  Google Scholar 

  30. Lampe, J., Voss, H.: Large-scale Tikhonov regularization of total least squares. J. Comput. Appl. Math. 238, 95–108 (2013)

    Article  MathSciNet  Google Scholar 

  31. Lawson, C. L., Hanson, R. J.: Solving Least Squares Problems, vol. 15. SIAM (1995)

  32. Lee, G., Barlow, J. L.: Two projection methods for regularized total least squares approximation. Linear Algebra Appl. 461, 18–41 (2014)

    Article  MathSciNet  Google Scholar 

  33. Lee, G., Fu, H., Barlow, J. L.: Fast high-resolution image reconstruction using Tikhonov regularization based total least squares. SIAM J. Sci. Comput. 35(1), B275–B290 (2013)

    Article  MathSciNet  Google Scholar 

  34. Levin, E., Meltzer, A. Y.: Estimation of the regularization parameter in linear discrete ill-posed problems using the Picard parameter. SIAM J. Sci. Comput. 39(6), A2741–A2762 (2017)

    Article  MathSciNet  Google Scholar 

  35. Markovsky, I.: Bibliography on total least squares and related methods. Stat. Interface 3(3), 329–334 (2010)

    Article  MathSciNet  Google Scholar 

  36. Markovsky, I., Van Huffel, S.: Overview of total least-squares methods. Signal Process. 87(10), 2283–2302 (2007)

    Article  Google Scholar 

  37. Miettinen, K.: Nonlinear Multiobjective Optimization, vol. 12. Springer Science & Business Media (2012)

  38. Nocedal, J., Wright, S.: Numerical optimization. Springer Science & Business Media (2006)

  39. Park, Y., Reichel, L., Rodriguez, G., Yu, X.: Parameter determination for Tikhonov regularization problems in general form. J. Comput. Appl. Math. 343, 12–25 (2018)

    Article  MathSciNet  Google Scholar 

  40. Reddy, G.: The parameter choice rules for weighted Tikhonov regularization scheme. Comput. Appl. Math. 37(2), 2039–2052 (2018)

    Article  MathSciNet  Google Scholar 

  41. Reichel, L., Rodriguez, G.: Old and new parameter choice rules for discrete ill-posed problems. Numer. Algorithm. 63(1), 65–87 (2013)

    Article  MathSciNet  Google Scholar 

  42. Reichel, L., Sadok, H.: A new L-curve for ill-posed problems. J. Comput. Appl. Math. 219(2), 493–508 (2008)

    Article  MathSciNet  Google Scholar 

  43. Renaut, R. A., Guo, H.: Efficient algorithms for solution of regularized total least squares. SIAM J. Matrix Anal. Appl. 26(2), 457–476 (2005)

    Article  MathSciNet  Google Scholar 

  44. Renaut, R. A., Horst, M., Wang, Y., Cochran, D., Hansen, J.: Efficient estimation of regularization parameters via downsampling and the singular value expansion. BIT Numer. Math. 57(2), 499–529 (2017)

    Article  Google Scholar 

  45. Sima, D. M., Van Huffel, S., Golub, G. H.: Regularized total least squares based on quadratic eigenvalue problem solvers. BIT Numer. Math. 44 (4), 793–812 (2004)

    Article  MathSciNet  Google Scholar 

  46. Snyman, J. A.: Practical mathematical optimization. Springer (2005)

  47. Van Huffel, S., Vandewalle, J.: The Total Least Squares Problem: Computational Aspects and Analysis, vol. 9. SIAM (1991)

  48. Wang, F., Zhao, X. -L., Ng, M. K.: Multiplicative noise and blur removal by framelet decomposition and l1-based L-curve method. IEEE Trans. Image Process. 25(9), 4222–4232 (2016)

    Article  MathSciNet  Google Scholar 

  49. Yang, M., Xia, Y., Wang, J., Peng, J.: Efficiently solving total least squares with Tikhonov identical regularization. Comput. Optim. Appl. 70(2), 571–592 (2018)

    Article  MathSciNet  Google Scholar 

  50. Zare, H., Hajarian, M.: Determination of regularization parameter via solving a multi-objective optimization problem. Appl. Numer. Math. 156, 542–554 (2020)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to express their heartfelt thanks to the editor and anonymous referees for their useful comments and constructive suggestions which substantially improved the quality and presentation of this paper.

Author information

Authors and Affiliations

  1. Faculty of Mathematical Sciences, Tarbiat Modares University, Tehran, Iran

    Hossein Zare

  2. Department of Applied Mathematics, Faculty of Mathematical Sciences, Shahid Beheshti University, Tehran, Iran

    Masoud Hajarian

Authors
  1. Hossein Zare
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masoud Hajarian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masoud Hajarian.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zare, H., Hajarian, M. An efficient Gauss–Newton algorithm for solving regularized total least squares problems. Numer Algor 89, 1049–1073 (2022). https://doi.org/10.1007/s11075-021-01145-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-021-01145-2

Keywords

Mathematics Subject Classification (2010)

Search

Navigation