Skip to main content
SpringerLink
Log in

Fast linear algebra is stable

  • Published:
Numerische Mathematik Aims and scope Submit manuscript

Abstract

In Demmel et al. (Numer. Math. 106(2), 199–224, 2007) we showed that a large class of fast recursive matrix multiplication algorithms is stable in a normwise sense, and that in fact if multiplication of n-by-n matrices can be done by any algorithm in O(n ω+η) operations for any η >  0, then it can be done stably in O(n ω+η) operations for any η >  0. Here we extend this result to show that essentially all standard linear algebra operations, including LU decomposition, QR decomposition, linear equation solving, matrix inversion, solving least squares problems, (generalized) eigenvalue problems and the singular value decomposition can also be done stably (in a normwise sense) in O(n ω+η) operations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson, E., Bai, Z., Bischof, C., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammarling, S., McKenney, A., Blackford, S., Sorensen, D.: LAPACK Users’ Guide, 3rd edn. SIAM, Philadelphia (1999)

  2. Anderson T.W., Olkin I. and Underhill L.G. (1987). Generation of random orthogonal matrices. SIAM J. Sci. Stat. Comput. 8: 625–629

    Article  MATH  MathSciNet  Google Scholar 

  3. Bai, Z., Demmel, J.: Design of a parallel nonsymmetric eigenroutine toolbox, Part I. In: Proceedings of the Sixth SIAM Conference on Parallel Processing for Scientific Computing. SIAM. Long version available as UC Berkeley Computer Science report all.ps.Z via anonymous ftp from tr-ftp.cs.berkeley.edu, directory (1993) pub/tech-reports/csd/csd-92-718

  4. Bai, Z., Demmel, J.: Using the matrix sign function to compute invariant subspaces. SIAM J. Mat. Anal. Appl. 19 (1), 1998. Technical report title “Design of a Parallel Nonsymmetric Eigenroutine Toolbox, Part II”

  5. Bai, Z., Demmel, J., Gu, M.: Inverse free parallel spectral divide and conquer algorithms for nonsymmetric eigenproblems. Numer. Math. 76, 279–308, (1997). UC Berkeley CS Division Report UCB//CSD-94-793, Feb 94

    Google Scholar 

  6. Bini D. and Lotti D. (1980). Stability of fast algorithms for matrix multiplication. Numer. Math. 36: 63–72

    Article  MATH  MathSciNet  Google Scholar 

  7. Bischof, C., Huss-Lederman, S., Sun, X., Tsao, A.: The PRISM project: Infrastructure and algorithms for parallel eigensolvers. In: Proceedings of the Scalable Parallel Libraries Conference, Mississippi State, Mississippi. IEEE Computer Society, 1993

  8. Bischof C. and Quintana-Orti G. (1998). Computing rank-revealing QR factorizations of dense matrices. ACM Trans. Math. Softw. 24(2): 226–253

    Article  MATH  MathSciNet  Google Scholar 

  9. Bischof, C., Van Loan, C.: The WY representation for products of Householder matrices. SIAM J. Sci. Stat. Comput. 8(1), (1987)

  10. Blackford, L.S., Choi, J., Cleary, A., E. D’Azevedo, Demmel, J., Dhillon, I., Dongarra, J., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R.C.: ScaLAPACK Users’ Guide. SIAM, Philadelphia (1997)

  11. Borodin A. and Munro I. (1975). The Computational Complexity of Algebraic and Numeric Problems. American Elsevier, Amsterdam

    MATH  Google Scholar 

  12. Brent, R.P.: Algorithms for matrix multiplication, Report CS 157. Computer Science Department, Stanford University (1970)

  13. Bulgakov A.Ya. and Godunov S.K. (1988). Circular dichotomy of the spectrum of a matrix. Siberian Math. J. 29(5): 734–744

    Article  MATH  MathSciNet  Google Scholar 

  14. Bürgisser P., Clausen M. and Shokrollahi M.A. (1997). Algebraic Complexity Theory. Springer, Berlin

    MATH  Google Scholar 

  15. Chan T. (1987). Rank revealing QR factorizations. Linear Algebra Appl. 88(89): 67–82

    MathSciNet  Google Scholar 

  16. Chandrasekaran S. and Ipsen I. (1994). On rank-revealing QR factorizations. SIAM J. Matrix Anal. Appl. 15(2): 592–622

    Article  MATH  MathSciNet  Google Scholar 

  17. Cohn, H., Kleinberg, R., Szegedy, B., Umans, C.: Group-theoretic algorithms for matrix multiplication. In: Foundations of Computer Science. 46th Annual IEEE Symposium on 23–25 October 2005, pp. 379–388 (2005)

  18. Cohn, H., Umans, C.: A group-theoretic approach to matrix multiplication. In: Foundations of Computer Science. 44th Annual IEEE Symposium, pp. 438–449 (2003)

  19. Coppersmith D. and Winograd S. (1990). Matrix multiplication via arithmetic progressions. J. Symbolic Comput. 9(3): 251–280

    Article  MATH  MathSciNet  Google Scholar 

  20. Cormen R., Leiserson C. and Rivest R. (1990). Algorithms. MIT/McGraw-Hill, Cambridge/New York

    MATH  Google Scholar 

  21. Day, D.: How the QR algorithm fails to converge and how to fix it. Technical Report 96-0913J, Sandia National Laboratory, April 1996

  22. Demmel, J.: Applied Numerical Linear Algebra. SIAM, 1997

  23. Demmel J., Dumitriu I., Holtz O. and Kleinberg R. (2007). Fast matrix multiplication is stable. Numer. Math. 106(2): 199–224

    Article  MATH  MathSciNet  Google Scholar 

  24. Demmel J. and Higham N.J. (1992). Stability of block algorithms with fast level 3 BLAS. ACM Trans. Math. Softw. 18: 274–291

    Article  MATH  Google Scholar 

  25. Demmel J., Higham N.J. and Schreiber R. (1995). Stability of block LU factorization, Numer. Linear Algebra Appl. 2(2): 173–190

    Article  MATH  MathSciNet  Google Scholar 

  26. Edelman A. (1988). Eigenvalues and condition numbers of random matrices. SIAM J. Math. Anal. Appl. 9(4): 543–560

    Article  MATH  MathSciNet  Google Scholar 

  27. Elmroth E. and Gustavson F. (2000). Applying recursion to serial and parallel QR factorization. IBM J. Res. Devel. 44(4): 605–624

    Article  Google Scholar 

  28. Godunov S.K. (1986). Problem of the dichotomy of the spectrum of a matrix. Siberian Math. J. 27(5): 649–660

    Article  MATH  MathSciNet  Google Scholar 

  29. Golub G. and Van Loan C. (1996). Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore

    MATH  Google Scholar 

  30. Gu M. and Eisenstat S. (1996). An efficient algorithm for computing a strong rank–revealing QR factorization. SIAM J. Sci. Comput. 17(4): 848–869

    Article  MATH  MathSciNet  Google Scholar 

  31. Heller D. (1978). A survey of parallel algorithms in numerical linear algebra. SIAM Rev. 20: 740–777

    Article  MATH  MathSciNet  Google Scholar 

  32. Higham N. (1995). Stability of parallel triangular matrix systems solvers. SIAM J. Sci. Comput. 16(2): 400–413

    Article  MATH  MathSciNet  Google Scholar 

  33. Higham N.J. (1990). Exploiting fast matrix multiplication within the Level 3 BLAS. ACM Trans. Math. Softw. 16: 352–368

    Article  MATH  MathSciNet  Google Scholar 

  34. Higham, N.J.: Accuracy and stability of numerical algorithms, 2nd edn. SIAM, Philadelphia, PA (2002)

  35. Hong P. and Pan C.T. (1992). Rank-revealing QR factorizations and the singular value decomposition. Math. Comput. 58: 213–232

    Article  MATH  MathSciNet  Google Scholar 

  36. Huss S., Quintana E.S., Sun X. and Wu J. (2000). Parallel spectral division using the matrix sign function for the generalized eigenproblem. Int. J. High Speed Comput. 11(1): 1–14

    Article  MATH  Google Scholar 

  37. Huss-Lederman, S., Tsao, A., Zhang, G.: A parallel implementation of the invariant subspace decomposition algorithm for dense symmetric matrices. In: Proceedings of the Sixth SIAM Conference on Parallel Processing for Scientific Computing. SIAM, 1993

  38. Malyshev A.N. (1989). Computing invariant subspaces of a regular linear pencil of matrices. Siberian Math. J. 30(4): 559–567

    Article  MATH  MathSciNet  Google Scholar 

  39. Malyshev, A.N.: Guaranteed accuracy in spectral problems of linear algebra, I,II. Siberian Adv. Math. 2 (1,2), 144–197, 153–204 (1992)

  40. Malyshev, A.N.: Parallel algorithm for solving some spectral problems of linear algebra. Linear Algebra Appl. 188,189, 489–520 (1993)

    Google Scholar 

  41. Muirhead R.J. (1982). Aspects of Multivariate Statistical Theory. Wiley, New York

    MATH  Google Scholar 

  42. PRISM: Parallel Research on Invariant Subspace Methods. www-unix.mcs.anl.gov/prism

  43. Ran Raz.: On the complexity of matrix product. SIAM J. Comput. 32(5), 1356–1369 (electronic), (2003)

  44. Roberts J. (1980). Linear model reduction and solution of the algebraic Riccati equation. Inter. J. Control 32: 677–687

    Article  MATH  Google Scholar 

  45. Schönhage A. and Strassen V. (1971). Schnelle Multiplikation grosser Zahlen. Computing (Arch. Elektron. Rechnen) 7: 281–292

    MATH  MathSciNet  Google Scholar 

  46. Schreiber R. and Van Loan C. (1989). A storage efficient WY representation for products of Householder transformations. SIAM J. Sci. Stat. Comput. 10: 53–57

    Article  MATH  Google Scholar 

  47. Stewart G.W. (1993). Updating a rank-revealing ULV decomposition. SIAM J. Math. Anal. Appl. 14(2): 494–499

    Article  MATH  Google Scholar 

  48. Stewart G.W. (1980). The efficient generation of random orthogonal matrices with an application to condition estimation. SIAM J. Numer. Anal. 17: 403–409

    Article  MATH  MathSciNet  Google Scholar 

  49. Strassen V. (1969). Gaussian elimination is not optimal. Numer. Math. 13: 354–356

    Article  MATH  MathSciNet  Google Scholar 

  50. Toledo S. (1997). Locality of reference in LU decomposition with partial pivoting. SIAM J. Math. Anal. Appl. 18(4): 1065–1081

    Article  MATH  MathSciNet  Google Scholar 

  51. Varah J. (1979). On the separation of two matrices. SIAM J. Numer. Anal. 16: 216–222

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematics Department and CS Division, University of California, Berkeley, CA, 94720, USA

    James Demmel

  2. Mathematics Department, University of Washington, Seattle, WA, 98195, USA

    Ioana Dumitriu

  3. Mathematics Department, University of California, Berkeley, CA, 94720, USA

    Olga Holtz

Authors
  1. James Demmel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ioana Dumitriu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olga Holtz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Olga Holtz.

Additional information

J. Demmel acknowledges support of NSF under grants CCF-0444486, ACI-00090127, CNS-0325873 and of DOE under grant DE-FC02-01ER25478.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Demmel, J., Dumitriu, I. & Holtz, O. Fast linear algebra is stable. Numer. Math. 108, 59–91 (2007). https://doi.org/10.1007/s00211-007-0114-x

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00211-007-0114-x

Mathematics Subject Classification (2000)

Search

Navigation