Abstract
This paper describes the performance characteristics of the LMS adaptive filter, a digital filter composed of a tapped delay line and adjustable weights, whose impulse response is controlled by an adaptive algorithm. For stationary stochastic inputs, the mean-square error, the difference between the filter output and an externally supplied input called the “desired response,” is a quadratic function of the weights, a paraboloid with a single fixed minimum point that can be sought by gradient techniques. The gradient estimation process is shown to introduce noise into the weight vector that is proportional to the speed of adaptation and number of weights. The effect of this noise is expressed in terms of a dimensionless quantity “misadjustment” that is a measure of the deviation from optimal Wiener performance. Analysis of a simple nonstationary case, in which the minimum point of the error surface is moving according to an assumed first-order Markov process, shows that an additional contribution to misadjustment arises from “lag” of the adaptive process in tracking the moving minimum point. This contribution, which is additive, is proportional to the number of weights but inversely proportional to the speed of adaptation. The sum of the misadjustments can be minimized by choosing the speed of adaptation to make equal the two contributions. It is further show, in Appendix A, that for stationary inputs the LMS adaptive algorithm, based on the method of steepest descent, approaches the theoretical limit of efficiency in terms of misadjustment and speed of adaptation when the eigenvalues of the input correlation matrix are equal or close in value. When the eigenvalues are highly disparate (λmax/λmin > 10), an algorithm similar to LMS but based on Newton’s method would approach this theoretical limit very closely.
Copyright 1976 by The Institute of Electrical and Electronics Engineers, Inc.; reprinted with permission from Proceedings of the IEEE, August 1976. This work was supported in part by the National Science Foundation under Grant ENGR 74-21752.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
B. Widrow and M. E. Hoff, “Adaptive switching circuits,” in 1960 WESCON Conv. Rec., pt. 4, pp. 96–140.
N. Nilsson, Learning Machines. New York: McGraw-Hill, 1965.
J. Koford and G. Groner, “The use of an adaptive threshold element to design a linear optimal pattern classifier,” IEEE Trans. Inform. Theory, vol. IT-12, pp. 42–50, Jan. 1966.
B. Widrow, P. Mantey, L. Griffiths, and B. Goode, “Adaptive antenna systems,” Proc. IEEE, vol. 55, pp. 2143–2159, Dec. 1967.
B. Widrow, “Adaptive Filters,” in Aspects of Network and System Theory, R. Kalman and N. DeClaris, Eds. New York: Holt, Rinehart, and Winston, 1971, pp. 563–587.
S. P. Applebaum, “Adaptive arrays,” Special Projects Lab., Syracuse Univ. Res. Corp., Rep. SPL 769.
L. J. Griffiths, “A simple adaptive algorithm for real-time processing in antenna arrays,” Proc. IEEE, vol. 57, pp. 1696–1704, Oct. 1969.
O. L. Frost III, “An algorithm for linearly constrained adaptive array processing,” Proc. IEEE, vol. 60, pp. 926–935, Aug. 1972.
W. F. Gabriel, “Adaptive arrays-An introduction,” Proc. IEEE, vol. 64, pp. 239–272, Feb. 1976.
F. W. Smith, “Design of quasi-optimal minimum-time controllers,” IEEE Trans. Automat. Contr., vol. AC-11, pp. 71–77, Jan. 1966.
B. Widrow, “Adaptive model control applied to real-time blood-pressure regulation,” in Pattern Recognition and Machine Learning, Proc. Japan-U.S. Seminar on the Learning Process in Control Systems, K. S. Fu, Ed. New York: Plenum Press, 1971, pp. 310–324.
R. Lucky, “Automatic equalization for digital communication,” Bell Syst. Tech. J., vol. 44, pp. 547–588, Apr. 1965.
M. DiToro, “A new method of high-speed adaptive serial com-munication through any time-variable and dispersive trans-mission medium,” in Conf. Record, 1965 IEEE Annual Communi-cations Convention, pp. 763–767.
R. Lucky and H. Rudin, “An automatic equalizer for general- purpose communication channels,” Bell Syst. Tech. J., vol. 46, pp. 2179–2208, Nov. 1967.
R. Lucky et al., Principles of Data Communication. New York: McGraw-Hill, 1968.
A. Gersho, “Adaptive equalization of highly dispersive channels for data transmission,” Bell Syst. Tech. J., vol. 48, pp. 55–70, Jan. 1969.
M. Soudhi, “An adaptive echo canceller,” Bell Syst. Tech. J., vol. 46, pp. 497–511, Mar. 1967.
B. Widrow et al., “Adaptive noise cancelling: Principles and applications,” Proc. IEEE, vol. 63, pp. 1692–1716, Dec. 1975.
P. E. Mantey, “Convergent automatic-synthesis procedures for sampled-data networks with feedback,” Stanford Electronics Laboratories, Stanford, CA, TR no. 7663 - 1, Oct. 1964.
P. M. Lion, “Rapid identification of linear and nonlinear systems,” in Proc. 1966 JACC, Seattle, WA, pp. 605–615, Aug. 1966; also AIAA Journal, vol. 5, pp. 1835–1842, Oct. 1967.
R. E. Ross and G. M. Lance, “An approximate steepest descent method for parameter identification,” in Proc. 1969 JACC, Boulder, CO, pp. 483–487, Aug. 1969.
R. Hastings-James and M. W. Sage, “Recursive generalized- least-squares procedure for online identification of process parameters,” Proc. IEE, vol. 116, pp. 2057–2062, Dec. 1969.
A. C. Soudack, K. L. Suryanarayanan, and S. G. Rao, “A uni¬fied approach to discrete-time systems identification,” Int. J. Control, vol. 14, no. 6, pp. 1009–1029, Dec. 1971.
W. Schaufelberger, “Der Entwurf adaptiver Systeme nach der direckten Methode von Ljapunov,” Nachrichtentechnik, Nr. 5, pp. 151–157, 1972.
J. M. Mendel, Discrete Techniques of Parameter Estimation: The Equation Error Formulation. New York: Marcel Dekker, Inc., 1973.
S. J. Merhav and E. Gabay, “Convergence properties in linear parameter tracking systems,” Identification and System Parameter Estimation-Part 2, Proc. 3rd IFAC Symp., P. Eyk- hoff, Ed. New York: American Elsevier Publishing Co., Inc., 1973, pp. 745–750.
R. V. Southwell, Relaxation Methods in Engineering Science. New York: Oxford, 1940.
D. J. Wilde, Optimum Seeking Methods. Englewood Cliffs, N.J.: Prentice-Hall, 1964.
B. Widrow, “Adaptive sampled-data systems,” in Proc. First Intern. Cong. Intern. Federation of Automatic Control, Moscow, 1960.
H. Robbins, and S. Monro, “A stochastic approximation method,” Ann. Math. Statist., vol. 22, pp. 400–407, 1951.
A. Dvoretzky, “On stochastic approximation,” in Proc. Third Berkeley Symp. Math. Statist, and Probability, J. Neyman, Ed. Berkeley, CA: University of California Press, 1956, pp. 39–55.
J. Makhoul, “Linear prediction: A tutorial review,” Proc. IEEE, vol. 63, pp. 561–580, Apr. 1975.
L. J. Griffiths, “Rapid measurement of digital instantaneous frequency,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-23, pp. 207–222, Apr. 1975.
Y. T. Chien, K. S. Fu, “Learning in non-stationary environment using dynamic stochastic approximation,” in Proc. 5th Allerton Conf. Circuit and Systems Theory, pp. 337–345, 1967.
T. P. Daniell and J. E. Brown III, “Adaptation in nonstation¬ary applications,” in Proc. 1970 IEEE Symp. Adaptive Processes (9th), Austin, TX, paper no. XXIV-4, Dec. 1970.
L. D. Davisson, “Steady-state error in adaptive mean-square minimization,” IEEE Trans. Inform. Theory, vol. IT-16, pp. 382–385, July 1970.
T. J. Schonfeld and M. Schwartz, “A rapidly converging first- order training algorithm for an adaptive equalizer,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 431–439, July 1971.
K. H. Mueller, “A new, fast-converging mean-square algorithm for adaptive equalizers with partial-response signaling,” Bell Syst. Tech. J., vol. 54, pp. 143–153, Jan. 1975.
L. J. Griffiths and P. E. Mantey, “Iterative least-squares algorithm for signal extraction,” in Proc. Second Hawaii Int. Conf. System Sciences, Western Periodicals Co., pp. 767–770, 1969.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1977 D. Reidel Publishing Company, Dordrecht-Holland
About this paper
Cite this paper
Widrow, B., McCool, J., Larimore, M.G., Johnson, C.R. (1977). Stationary and Nonstationary Learning Characteristics of the LMS Adaptive Filter. In: Tacconi, G. (eds) Aspects of Signal Processing. NATO Advanced Study Institutes Series, vol 33-1. Springer, Dordrecht. https://doi.org/10.1007/978-94-010-1223-2_23
Download citation
DOI: https://doi.org/10.1007/978-94-010-1223-2_23
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-010-1225-6
Online ISBN: 978-94-010-1223-2
eBook Packages: Springer Book Archive