Christian W. Omlin
Abstract
Recurrent neural networks that are trained to behave like deterministic finite-state automata (DFAs) can show deteriorating performance when tested on long strings. This deteriorating performance can be attributed to the instability of the internal representation of the learned DFA states. The use of a sigmoidel discriminant function together with the recurrent structure contribute to this instability. We prove that a simple algorithm can construct second-order recurrent neural networks with a sparse interconnection topology and sigmoidal discriminant function such that the internal DFA state representations are stable, that is, the constructed network correctly classifies strings of arbitrary length. The algorithm is based on encoding strengths of weights directly into the neural network. We derive a relationship between the weight strength and the number of DFA states for robust string classification. For a DFA with n state and minput alphabet symbols, the constructive algorithm generates a “programmed” neural network with O(n) neurons and O(mn) weights. We compare our algorithm to other methods proposed in the literature.
- ~ALON, N., DEWDNEY, A. K., AND OTT, T.J. 1991. Efficient simulation of finite automata by neural ~nets, JACM 38, 2 (Apr.), 495-514. Google Scholar
- ~BARNSLEY, M. 1988. Fractals Everywhere. Academic Press, San Diego, Calif. Google Scholar
- ~CASE~, M. 1996. The dynamics of discrete-time computation, with application to recurrent neural ~networks and finite state machine extraction, Neural Comput. 8, 6, 1135-1178. Google Scholar
- ~ELMAN, J. 1990. Finding structure in time. Cogn. Sci. 14, 179-211.Google Scholar
- ~FRASCONI, P., GORI, M., MAGGINI, M., AND SODA, G. 1996. Representation of finite state ~automata in recurrent radial basis function networks, Mach. Learn. 23, 5-32. Google Scholar
- ~FRASCONI, P., GORI, M., MAGGINI, M., AND SODA, G. 1991. A unified approach for integrating ~explicit knowledge and learning by example in recurrent networks. In Proceedings of the Interna- ~tional Joint Conference on Neural Networks, vol. 1. IEEE, New York, p. 811.Google Scholar
- ~FRASCONI, P., GORI, M., AND SODA, G. 1993. Injecting nondeterministic finite state automata into ~recurrent networks. Tech. Rep. Dipartimento di Sistemi e Informatica, Universit~ di Firenze, Italy, ~Florence, Italy.Google Scholar
- ~GEMAN, S., BIENENSTOCK, E., AND DOURSTAT, R. 1992. Neural networks and the bias/variance ~dilemma, Neural Comput. 4, 1, 1-58. Google Scholar
- ~GILES, C., CHEN, D., MILLER, C., CHEN, H., SUN, G., AND LEE, Y. 1991. Second-order recurrent ~neural networks for grammatical inference. In Proceedings of the International Joint Conference on ~Neural Networks 1991, vol. II. IEEE, New York, pp. 273-281.Google Scholar
- ~GILES, C., KUHN, G., AND WILLIAMS, R. 1994. Special issue on Dynamic recurrent neural networks: ~Theory and applications, IEEE Trans. Neural Netw. 5, 2.Google Scholar
- ~GILES, C., MILLER, C., CHEN, D., CHEN, H., SUN, G., AND LEE, Y. 1992. Learning and extracting ~finite state automata with second-order recurrent neural networks. Neural Comput. 4, 3, 380. Google Scholar
- ~GILES, C., AND OMLIN, C. 1992. Inserting rules into recurrent neural networks. In NeuralNetworks ~for Signal Processing II, Proceedings of the 1992 IEEE Workshop (S. Kung, F. Fallside, J. A. ~Sorenson, and C. Kamm, eds.) IEEE, New York, pp. 13-22.Google Scholar
- ~GILES, C., AND OMLIN, C. 1993. Rule refinement with recurrent neural networks. In Proceedings ~IEEE International Conference on Neural Networks (ICNN'93), vol. II. IEEE, New York, pp. ~801-806.Google Scholar
- HAYKIN, S. 1994. Neural Networks, A Comprehensive Foundation. MacMillan, New York. Google Scholar
- ~HIRSCH, M. 1989. Convergent activation dynamics in continuous-time neural networks. Neural ~Netw. 2, 331-351. Google Scholar
- ~HIRSCH, M. 1994. Saturation at high gain in discrete time recurrent networks. Neural Netw. 7, 3, ~449 -453. Google Scholar
- ~HOPCROFT, J., AND ULLMAN, J. 1979. Introduction to Automata Theory, Languages, and Computa- ~tion. Addison-Wesley, Reading, Mass. Google Scholar
- ~HORNE, B., AND HUSH, D. 1996. Bounds on the complexity of recurrent neural network implemen- ~tations of finite state machines. Neural Netw. 9, 2, 243-252. Google Scholar
- ~MACLIN, R., AND SHAVLIK, J. 1993. Using knowledge-based neural networks to improve algo- ~rithms: Refining the Chou-Fasman algorithm for protein folding. Mach. Learn. 11, 195-215. Google Scholar
- MEAD, C. 1989. Analog VLSI and Neural Systems. Addison-Wesley, Reading, Mass. Google Scholar
- ~MINSKY, M. 1967. Computation: Finite and Infinite Machines. Prentice-Hall, Inc., Englewood Cliffs, ~N.J., pp. 32-66 (Chap. 3). Google Scholar
- ~OMLIN, C., AND GILES, C. 1996a. Rule revision with recurrent neural networks. IEEE Trans. ~Knowl. Data Eng. 8, 1, 183-188. Google Scholar
- ~OMLIN, C., AND GILES, C. 1996b. Stable encoding of large finite-state automata in recurrent neural ~networks with sigmoid discriminants. Neural Comput. 8, 4, 675-696. Google Scholar
- ~OMLIN, C., AND GILES, C. 1992. Training second-order recurrent neural networks using hints, in ~Proceedings of the 9th International Conference on Machine Learning (San Mateo, Calif.), D. ~Sleeman and P. Edwards, eds. Morgan-Kaufmann, San Mateo, Calif., pp. 363-368. Google Scholar
- POLLACK, J. 1991. The induction of dynamical recognizers. Mach. Learn. 7, 227-252. Google Scholar
- SERVAN-SCHREIBER, D., CLEEREMANS, A., AND MCCLELLAND, J. 1991. Graded state machine: The ~representation of temporal contingencies in simple recurrent networks. Mach. Learn. 7, 161. Google Scholar
- ~SHAVLIK, J. 1994. Combining symbolic and neural learning. Mach. Learn. 14, 3, 321-331. Google Scholar
- ~SHEU, B.J. 1995. Neural Information Processing and VLSI. Kluwer Academic Publishers, Boston, ~Mass. Google Scholar
- ~TINO, P., HORNE, B. AND GLEES, C. 1995. Fixed points in two-neuron discrete time recurrent ~networks: Stability and bifurcation considerations. Tech. Rep. UMIACS-TR-95-51. Institute for ~Advanced Computer Studies, Univ. Maryland, College Park, Md. Google Scholar
- ~TOWELL, G., SHAVLIK, J., AND NOORDEWIER, M. 1990. Refinement of approximately correct ~domain theories by knowledge-based neural networks. In Proceedings of the 8th National Conference ~on Artificial Intelligence (San Mateo, Calif.) Morgan-Kaufmann, San Mateo, Calif., p. 861.Google Scholar
- ~WATROUS, R., AND KUHN, G. 1992. Induction of finite-state languages using second-order recur- ~rent networks. Neural Comput. 4, 3, 406. Google Scholar
- ~ZENG, Z., GOODMAN, R., AND SMYTH, P. 1993. Learning finite state machines with self-clustering ~recurrent networks. Neural Comput. 5, 6, 976-990. Google Scholar
Index Terms
- Constructing deterministic finite-state automata in recurrent neural networks
Recommendations
Nonlinear enhancement of noisy speech, using continuous attractor dynamics formed in recurrent neural networks
Here, formation of continuous attractor dynamics in a nonlinear recurrent neural network is used to achieve a nonlinear speech denoising method, in order to implement robust phoneme recognition and information retrieval. Formation of attractor dynamics ...
First-order recurrent neural networks and deterministic finite state automata
We examine the correspondence between first-order recurrent neural networks and deterministic finite state automata. We begin with the problem of inducing deterministic finite state automata from finite training sets, that include both positive and ...
Expressive power of first-order recurrent neural networks determined by their attractor dynamics
We characterize the attractor-based expressive power of several models of recurrent neural networks.The deterministic rational-weighted networks are Muller Turing equivalent.The deterministic real-weighted and evolving networks recognize the class of B ...
Comments