Abstract
Numerical and computational aspects of direct methods for largeand sparseleast squares problems are considered. After a brief survey of the most oftenused methods, we summarize the important conclusions made from anumerical comparison in matlab. Significantly improved algorithms haveduring the last 10-15 years made sparse QR factorization attractive, andcompetitive to previously recommended alternatives. Of particular importanceis the multifrontal approach, characterized by low fill-in, dense subproblemsand naturally implemented parallelism. We describe a Householder multifrontalscheme and its implementation on sequential and parallel computers. Availablesoftware has in practice a great influence on the choice of numericalalgorithms. Less appropriate algorithms are thus often used solely because ofexisting software packages. We briefly survey softwarepackages for the solution of sparse linear least squares problems. Finally,we focus on various applications from optimization, leading to the solution oflarge and sparse linear least squares problems. In particular, we concentrateon the important case where the coefficient matrix is a fixed general sparsematrix with a variable diagonal matrix below. Inner point methods forconstrained linear least squares problems give, for example, rise to suchsubproblems. Important gains can be made by taking advantage of structure.Closely related is also the choice of numerical method for these subproblems.We discuss why the less accurate normal equations tend to be sufficient inmany applications.
Similar content being viewed by others
References
M. Arioli, I. S. Duff, and P. de Rijk, On the augmented system approach to sparse least-squares problems, Numer. Math., 55 (1989), pp. 667–684.
Å. Björck, Stability analysis of the method of semi-normal equations for least squares problems, Linear Algebra Appl., 88/89 (1987), pp. 31–48.
Å. Björck, A direct method for sparse least squares problems with lower and upper bounds, Numer. Math., 54 (1988), pp. 19–32.
Å. Björck, Pivoting and stability in the augmented system method, Technical Report LiTH-MAT-R-1991-30, Department of Mathematics, Linköping University, June 1991.
Å. Björck, Numerical Methods for Least Squares Problems, SIAM, Philadelphia, 1996.
J. R. Bunch and L. Kaufman, Some stable methods for calculating inertia and solving symmetric linear systems, Mathematics of Computation, 31 (1977), pp. 162–179.
E. C. H. Chu, J. A. George, J. Liu, and E. Ng, SPARSPAK: Waterloo sparse matrix package user's guide for SPARSPAK-A, Research Report CS-84-36, Dept. of Computer Science, University of Waterloo, 1984.
T. F. Coleman, A. Edenbrandt, and J. R. Gilbert, Predicting fill for sparse orthogonal factorization, J. ACM, 33 (1986), pp. 517–532.
J. Dennis and R. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, Englewood Cliffs, N.J., 1983.
I. S. Duff, Pivot selection and row orderings in Givens reduction on sparse matrices, Computing, 13 (1974), pp. 239–248.
I. S. Duff, N. I. M. Gould, J. K. Reid, J. A. Scott, and K. Turner, The factorization of sparse symmetric indefinite matrices, IMA J. Numer. Anal., 11 (1991), pp. 181–204.
I. S. Duff, R. G. Grimes, and J. G. Lewis, Sparse matrix test problems, ACMTrans. Math. Softw., 15 (1989), pp. 1–14.
I. S. Duff and J. K. Reid, A comparison of some methods for the solution of sparse overdetermined systems of linear equations, J. Inst. Maths. Applics., 17 (1976), pp. 267–280.
I. S. Duff and J. K. Reid, The multifrontal solution of indefinite sparse symmetric linear systems, ACM Trans. Math. Software, 9 (1983), pp. 302–325.
S. C. Eisenstat, M. H. Schultz, and A. H. Sherman, Algorithms and data structures for sparse symmetric Gaussian elimination, SIAM J. Sci. Statist. Comput., 2 (1981), pp. 225–237.
L. EldÉn, Algorithms for the regularization of ill-conditioned least squares problems, BIT, 17 (1977), pp. 134–145.
L. V. Foster, Modifications of the normal equations method that are numerically stable, in Numerical Linear Algebra, Digital Signal Processing and Parallel Algorithms, G. H. Golub and </del> P. V. Dooren, eds., NATO ASI Series, Berlin, 1991, Springer-Verlag, pp. 501–512.
C. F. Gauss, Theory of the Motion of the Heavenly Bodies Moving about the Sun in Conic Sections, Dover, New York (1963), 1809. C. H. Davis, Trans.
W. M. Gentleman, Basic procedures for large, sparse, or weighted linear least squares problems, Research report CSRR 2068, University of Waterloo, Waterloo, Ontario, Canada, July 1972.
W. M. Gentleman, Least squares computations by Givens transformations without square roots, J. Inst. Maths. Applics., 12 (1973), pp. 329–336.
W. M. Gentleman, Error analysis of QR decompositions by Givens transformations, Linear Algebra Appl., 10 (1975), pp. 189–197.
W. M. Gentleman, Row elimination for solving sparse linear systems and least squares problems, in Proceedings the 6th Dundee Conference on Numerical Analysis, G. A. Watson, ed., Springer Verlag, 1976, pp. 122–133.
J. A. George and M. T. Heath, Solution of sparse linear least squares problems using Givens rotations, Linear Algebra Appl., 34 (1980), pp. 69–83.
J. A. George and J. W.-H. Liu, Householder reflections versus Givens rotations in sparse orthogonal decomposition, Linear Algebra Appl., 88/89 (1987), pp. 223–238.
J. A. George and E. G. Ng, On row and column orderings for sparse least squares problems, SIAM J. Numer. Anal., 20 (1981), pp. 326–344.
J. A. George and E. G. Ng, SPARSPAK:Waterloo sparse matrix package user's guide for SPARSPAKB, Research Report CS-84-37, Dept. of Computer Science, University of Waterloo, 1984.
J. R. Gilbert, C. Moler, and R. Schreiber, Sparse matrices in MATLAB: Design and implementation, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 333–356.
P. E. Gill and W. Murray, Nonlinear least squares and nonlinearly constrained Optimization, in In Proceedings Dundee Conference on Numerical Analysis 1975, Lecture Notes in Mathematics No. 506, Springer Verlag, 1976.
P. E. Gill and W. Murray, The orthogonal factorization of a large sparse matrix, in Sparse Matrix Computations, J. R. Bunch and D. J. Rose, eds., New York, 1976, Academic Press, pp. 201–212.
P. E. Gill, W. Murray, and M. H. Wright, Practical Optimization, Academic Press, London and New York, 1981.
G. H. Golub, Numerical methods for solving least squares problems, Numer. Math., 7 (1965), pp. 206–216.
C. C. Gonzaga, Path-following methods for linear programming, SIAM Review, 34 (1992), pp. 167–224.
M. T. Heath, Numerical methods for large sparse linear least squares problems, SIAM J. Sci. Statist. Comput.,, 5 (1984), pp. 497–513.
G. B. Kolata, Geodesy: Dealing with an enormous computer task, Science, 200 (1978), pp. 421–422.
C. L. Lawson and R. J. Hanson, Solving Least Squares Problems, Prentice Hall, Englewood Cliffs, New Jersey, 1974.
A. M. Legendre, Nouvelle méthodes pour la détermination des orbites des comètes, Courcier, Paris, 1805.
K. Levenberg, A method for the solution of certain non-linear problems in least squares, Quart. Appl. Math., 2 (1944), pp. 164–168.
J. W.-H. Liu, On general row merging schemes for sparse Givens transformations, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 1190–1211.
J. W.-H. Liu, The role of elimination trees in sparse factorization, SIAM J. Matrix Anal. Appl., 11 (1990), pp. 134–172.
S. Lu and J. L. Barlow, Multifrontal computation with the orthogonal factors of sparse matrices, SIAM J. Matr. Anal. 3(1996), pp. 658–679.
I. Lustig, R. Marsten, and D. Shanno, Computational experience with a primal-dual interior point method for linear programming, Linear Algebra Appl., (1991), pp. 191–222.
D. W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society for Industrial and Applied Mathematics, 11 (1963), pp. 431–441.
P. Matstoms, The multifrontal solution of sparse linear least squares problems, Licentiat thesis, Linköping University, 1991.
P. Matstoms, QR27-Specification sheet, Tech. Report March 1992, Department of Mathematics, 1992.
P. Matstoms, SparseQRfactorization inMATLAB, ACMTrans. Math. Software, 20 (1994), pp. 136–159.
P. Matstoms, Sparse QR Factorization with Applications to Linear Least Squares Problems, PhD thesis, Linköping University, 1994.
P. Matstoms, Parallel sparse QR factorization on shared memory architectures, Parallel Computing, 21 (1995), pp. 473–486.
J. J. MorÉ, The Levenberg-Marquardt algorithm: Implementation and theory, in G.A. Watson, Lecture Notes in Math. 630, Berlin, 1978, Springer Verlag, pp. 105–116.
U. Oreborn, A Direct Method for Sparse Nonnegative Least Squares Problems, licentiat thesis, Linköping University, 1986.
C. C. Paige and M. A. Saunders, LSQR. an algorithm for sparse linear equations and sparse least squares, ACM Trans. Math. Software, 8 (1982), pp. 43–71.
D. J. Pierce and J. G. Lewis, Sparse multifrontal rank revealing QR factorization, Technical Report MEA-TR-193-Revised, Boeing Information and Support Services, 1995.
L. F. Portugal, J. J. JÚdice, and L. N. Vicente, Solution of large scale linear least-squares problems with nonnegativ variables, technical report, Departamento de ciêcias da terra, Universidade de Coimbra, 3000 Coimbra, Portugal, 1993.
C. Puglisi, QR factorization of large sparse overdetermined and square matrices with the multifrontal method in a multiprocessor environment, PhD thesis, CERFACS, 42 av. G. Coriolis, 31057 Toulouse Cedex, France.
J. K. Reid, A note on the least squares solution of a band system of linear equations by Householder reductions, Comput J., 10 (1967), pp. 188–189.
C. Sun, Parallel sparse orthogonal factorization on distributed-memory multiprocessors, SIAM J. Sci. Comput., 17 (1996), p. to appear.
M. H. Wright, Interior methods for constrained Optimization, Acta Numerica 1992, Cambridge University Press, 1992, pp. 341–407.
Z. Zlatev and H. Nielsen, LLSS01-a Fortran subroutine for solving least squares problems ( User's guide ), Technical Report 79-07, Institute of Numerical Analysis, Technical University of Denmark, Lyngby, Denmark, 1979.
Z. Zlatev and H. Nielsen, Solving large and sparse linear least-squares problems by conjugate gradient algorithms, Comput. Math. Applic., 15 (1988), pp. 185–202.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Matstoms, P. Sparse Linear Least Squares Problems in Optimization. Computational Optimization and Applications 7, 89–110 (1997). https://doi.org/10.1023/A:1008680131271
Issue Date:
DOI: https://doi.org/10.1023/A:1008680131271