Skip to main content
SpringerLink
Log in

Linear Convergence of Iterative Soft-Thresholding

  • Published:
Journal of Fourier Analysis and Applications Aims and scope Submit manuscript

Abstract

In this article a unified approach to iterative soft-thresholding algorithms for the solution of linear operator equations in infinite dimensional Hilbert spaces is presented. We formulate the algorithm in the framework of generalized gradient methods and present a new convergence analysis. As main result we show that the algorithm converges with linear rate as soon as the underlying operator satisfies the so-called finite basis injectivity property or the minimizer possesses a so-called strict sparsity pattern. Moreover it is shown that the constants can be calculated explicitly in special cases (i.e. for compact operators). Furthermore, the techniques also can be used to establish linear convergence for related methods such as the iterative thresholding algorithm for joint sparsity and the accelerated gradient projection method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bauschke, H.H., Borwein, J.M., Combettes, P.L.: Bregman monotone optimization algorithms. SIAM J. Control Optim. 42(2), 596–636 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  2. Bredies, K., Lorenz, D.A.: Iterated hard shrinkage for minimization problems with sparsity constraints. SIAM J. Sci. Comput. 30(2), 657–683 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  3. Bredies, K., Lorenz, D.A., Maass, P.: A generalized conditional gradient method and its connection to an iterative shrinkage method. Comput. Optim. Appl. (2008). doi:10.1007/s10589-007-9083-3

  4. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  5. Dahlke, S., Maass, P.: An outline of adaptive wavelet Galerkin methods for Tikhonov regularization of inverse parabolic problems. In: Hon, J.C.Y.-C., Yamamoto, M., Lee, J.-Y. (eds.) Recent Development in Theories & Numerics, pp. 56–66. World Scientific, Singapore (2003)

    Chapter  Google Scholar 

  6. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure App. Math. 57(11), 1413–1457 (2004)

    Article  MATH  Google Scholar 

  7. Daubechies, I., Fornasier, M., Loris, I.: Accelerated projected gradient method for linear inverse problems with sparsity constraints. J. Fourier Anal. Appl. (2008), this issue. doi:10.1007/s00041-008-9039-8

  8. Demyanov, V.F., Rubinov, A.M.: Approximate Methods in Optimization Problems. Modern Analytic and Computational Methods in Science and Mathematics, vol. 32. American Elsevier, New York (1970)

    Google Scholar 

  9. Donoho, D.L., Elad, M., Temlyakov, V.: Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory 52, 6–18 (2006)

    Article  MathSciNet  Google Scholar 

  10. Duarte, M.F., Sarvotham, S., Baron, D., Wakin, M.B., Baraniuk, R.: Distributed compressed sensing of jointly sparse signals. In: Conference Record of the Thirty-Ninth Asilomar Conference on Signals, Systems And Computers, pp. 1537–1541 (2005)

  11. Dunn, J.C.: Global and asymptotic convergence rate estimates for a class of projected gradient processes. SIAM J. Control Optim. 19(3), 368–400 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  12. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32(2), 407–451 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  13. Ekeland, I., Temam, R.: Convex Analysis and Variational Problems. North-Holland, Amsterdam (1976)

    MATH  Google Scholar 

  14. Elad, M., Matalon, B., Zibulevsky, M.: Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization. Appl. Comput. Harmon. Anal. 23, 346–367 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  15. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Mathematics and its Applications, vol. 375. Kluwer Academic, Dordrecht (2000)

    Google Scholar 

  16. Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: Applications to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 4, 586–597 (2007)

    Article  Google Scholar 

  17. Fornasier, M., Rauhut, H.: Recovery algorithms for vector valued data with joint sparsity constraints. SIAM J. Numer. Anal. 46(2), 577–613 (2008)

    Article  MathSciNet  Google Scholar 

  18. Fuchs, J.-J.: Recovery of exact sparse representations in the presence of bounded noise. IEEE Trans. Inf. Theory 51(10), 3601–3608 (2005)

    Article  Google Scholar 

  19. Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70, 709–710 (1964)

    Article  MATH  Google Scholar 

  20. Gribonval, R., Nielsen, M.: Highly sparse representations from dictionaries are unique and independent of the sparseness measure. Appl. Comput. Harmon. Anal. 22(3), 335–355 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  21. Griesse, R., Lorenz, D.A.: A semismooth Newton method for Tikhonov functionals with sparsity constraints. Inverse Probl. 24, 035007 (2008)

    Article  MathSciNet  Google Scholar 

  22. Kim, S.-J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: A method for large-scale ℓ 1-regularized least squares problems with applications in signal processing and statistics. IEEE J. Sel. Top. Signal Process. 4, 606–617 (2007)

    Article  Google Scholar 

  23. Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966)

    Article  Google Scholar 

  24. Lorenz, D.A.: Convergence rates and source conditions for Tikhonov regularization with sparsity constraints. J. Inverse Ill-Posed Probl. (2008, to appear)

  25. Maass, P.: The interior radon transform. SIAM J. Appl. Math. 52(3), 710–724 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  26. Nachman, A.I.: Global uniqueness for a two-dimensional inverse boundary value problem. Ann. Math. 143, 71–96 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  27. Osborne, M.R., Presnell, B., Turlach, B.A.: A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20, 389–404 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  28. Resmerita, E.: Regularization of ill-posed problems in Banach spaces: convergence rates. Inverse Probl. 21(4), 1303–1314 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  29. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, New York (1998)

    MATH  Google Scholar 

  30. Schöpfer, F., Louis, A.K., Schuster, T.: Nonlinear iterative methods for linear ill-posed problems in Banach spaces. Inverse Probl. 22, 311–329 (2006)

    Article  MATH  Google Scholar 

  31. Showalter, R.E.: Monotone Operators in Banach Space and Nonlinear Partial Differential Equations. Mathematical Surveys and Monographs, vol. 49. American Mathematical Society, Providence (1997)

    Google Scholar 

  32. Teschke, G., Ramlau, R.: An iterative algorithm for nonlinear inverse problems with joint sparsity constraints in vector-valued regimes and an application to color image inpainting. Inverse Probl. 23(5), 1851–1870 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  33. Turlach, B.A.: On algorithms for solving least squares problems under an L 1 penalty or an L 1 constraint. In: 2004 Proceedings of the American Statistical Association, Statistical Computing Section, pp. 2572–2577. American Statistical Association, New York (2005)

    Google Scholar 

  34. Xu, Z.B., Roach, G.F.: Characteristic inequalities of uniformly convex and uniformly smooth Banach spaces. J. Math. Anal. Appl. 157(1), 189–210 (1991)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Industrial Mathematics / Fachbereich 3, University of Bremen, Postfach 33 04 40, 28334, Bremen, Germany

    Kristian Bredies & Dirk A. Lorenz

Authors
  1. Kristian Bredies
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dirk A. Lorenz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dirk A. Lorenz.

Additional information

Communicated by Michael Elad.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bredies, K., Lorenz, D.A. Linear Convergence of Iterative Soft-Thresholding. J Fourier Anal Appl 14, 813–837 (2008). https://doi.org/10.1007/s00041-008-9041-1

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00041-008-9041-1

Keywords

Mathematics Subject Classification (2000)

Search

Navigation