Matrix Computations and Neural Associative Memories

Abstract

In view of recent interest and applications of Neural Associative Memories, it becomes increasingly important to evaluate their capacity limits and saturation effects. This paper presents a unified mathematical approach to evaluation of saturation/capacity for a large class of associative memories based upon matrix operations. This class includes, among others, Correlation Matrix Memory, Higher Order Associative Memory, Generalized Inverse Memory and Hamming net. The general model is based on Linear Algebra and is applicable to both binary and continuous-valued memories, and also includes auto-associative, hetero-associative and classification modes of operation. It is argued and demonstrated that the well-understood Linear Algebra formalism can be effectively applied to evaluate saturation/capacity limits, scaling properties and various input/output encoding schemes, as well as to compare different supervised learning (memory construction) techniques. As a practical application of our approach, we present detailed comparative analysis of the Outer Product Learning and the Generalized Inverse Memory construction rules for the auto-associative memory and the unary classification modes of operation.