Abstract
Computational mechanics quantifies structure in a stochastic process via its causal states, leading to the process’s minimal, optimal predictor—the \(\epsilon {\text {-}}\mathrm{machine}\). We extend computational mechanics to communication channels coupling two processes, obtaining an analogous optimal model—the \(\epsilon {\text {-}}\mathrm{transducer}\)—of the stochastic mapping between them. Here, we lay the foundation of a structural analysis of communication channels, treating joint processes and processes with input. The result is a principled structural analysis of mechanisms that support information flow between processes. It is the first in a series on the structural information theory of memoryful channels, channel composition, and allied conditional information measures.
Similar content being viewed by others
References
Crutchfield, J.P.: Between order and chaos. Nat. Phys. 8(January), 17–24 (2012)
Packard, N.H., Crutchfield, J.P., Farmer, J.D., Shaw, R.S.: Geometry from a time series. Phys. Rev. Lett. 45, 712 (1980)
Mandal, D., Jarzynski, C.: Work and information processing in a solvable model of Maxwell’s Demon. Proc. Natl. Acad. Sci. USA 109(29), 11641–11645 (2012)
Boyd, A.B., Crutchfield, J.P.: Demon dynamics: deterministic chaos, the Szilard map, and the intelligence of thermodynamic systems. Santa Fe Institute Working Paper 15–06-019. arXiv:1506.04327 [cond-mat.stat-mech]
Boyd, A.B., Mandal, D., Crutchfield, J.P.: Identifying functional thermodynamics in autonomous Maxwellian ratchets. Santa Fe Institute Working Paper 15-07-025. arXiv:1507.01537 [cond-mat.stat-mech]
Rieke, F., Warland, D., de Ruyter, R., van Steveninck, Bialek, W.: Spikes: Exploring the Neural Code. Bradford Book, New York (1999)
Cutsuridis, V., Hussain, A., Taylor, J.G.: Perception-Action Cycle. Springer, New York (2011)
Gordon, G., Kaplan, D., Lankow, D., Little, D., Sherwin, J., Suter, B., et al.: Toward an integrated approach to perception and action: conference report and future directions. Front. Syst. Neurosci. 5, 20 (2011)
Padgett, J.F., Lee, D., Collier, N.: Economic production as chemistry. Ind. Corp. Chang. 12(4), 843–877 (2003)
Crutchfield, J.P., Young, K.: Inferring statistical complexity. Phys Rev. Lett. 63, 105–108 (1989)
Gray, R.M.: Probability, Random Processes, and Ergodic Theory, 2nd edn. Springer, New York (2009)
Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley, New York (2006)
Yeung, R.W.: Information Theory and Network Coding. Springer, New York (2008)
Ellison, C.J., Mahoney, J.R., Crutchfield, J.P.: Prediction, retrodiction, and the amount of information stored in the present. J. Stat. Phys. 136(6), 1005–1034 (2009)
Crutchfield, J.P., Ellison, C.J., Mahoney, J.R., James, R.G.: Synchronization and control in intrinsic and designed computation: an information-theoretic analysis of competing models of stochastic computation. CHAOS 20(3), 037105 (2010)
Gray, R.M., Davisson, L.D.: The ergodic decomposition of stationary discrete random processses. IEEE Trans. Inf. Theory 20(5), 625–636 (1974)
Billingsley, P.: Statistical methods in Markov chains. Ann. Math. Stat. 32, 12 (1961)
Crutchfield, J.P., Shalizi, C.R.: Thermodynamic depth of causal states: objective complexity via minimal representations. Phys. Rev. E 59(1), 275–283 (1999)
Shalizi, C.R., Crutchfield, J.P.: Computational mechanics: pattern and prediction, structure and simplicity. J. Stat. Phys. 104, 817–879 (2001)
Crutchfield, J.P., Ellison, C.J., Mahoney, J.R.: Time’s barbed arrow: irreversibility, crypticity, and stored information. Phys. Rev. Lett. 103(9), 094101 (2009)
N. Travers and J. P. Crutchfield. Equivalence of history and generator \(\epsilon \)-machines. Santa Fe Institute Working Paper 11–11-051. arXiv:1111.4500 [math.PR]
Crutchfield, J.P.: The calculi of emergence: computation, dynamics, and induction. Physica D 75, 11–54 (1994)
Paz, A.: Introduction to Probabilistic Automata. Academic Press, New York (1971)
Hopcroft, J.E., Ullman, J.D.: Introduction to Automata Theory, Languages, and Computation. Addison-Wesley, Reading (1979)
Lind, D., Marcus, B.: An Introduction to Symbolic Dynamics and Coding. Cambridge University Press, New York (1995)
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(379–423), 623–656 (1948)
Gray, R.M.: Entropy and Information Theory. Springer, New York (1990)
Oppenheim, A.V., Schafer, R.W.: Discrete-Time Signal Processing. Prentice-Hall Signal Processing Series. Prentice Hall, Upper Saddle River (2010)
Mandelbrot, B.: Fractals: Form. Chance and Dimension. W. H. Freeman and Company, New York (1977)
Rabiner, L.R.: A tutorial on hidden Markov models and selected applications. IEEE Proc. 77, 257 (1989)
Elliot, R.J., Aggoun, L., Moore, J.B.: Hidden Markov Models: Estimation and Control. Applications of Mathematics, vol. 29. Springer, New York (1995)
James, R.G., Mahoney, J.R., Ellison, C.J., Crutchfield, J.P.: Many roads to synchrony: natural time scales and their algorithms. Phys. Rev. E 89, 042135 (2014)
Crutchfield, J.P., Hanson, J.E.: Turbulent pattern bases for cellular automata. Physica D 69, 279–301 (1993)
Li, C.-B., Yang, H., Komatsuzaki, T.: Multiscale complex network of protein conformational fluctuations in single-molecule time series. Proc. Natl. Acad. Sci. USA 105, 536–541 (2008)
Varn, D.P., Canright, G.S., Crutchfield, J.P.: \(\epsilon \)-Machine spectral reconstruction theory: a direct method for inferring planar disorder and structure from X-ray diffraction studies. Acta. Cryst. Sec. A 69(2), 197–206 (2013)
Strelioff, C.C., Crutchfield, J.P.: Bayesian structural inference for hidden processes. Phys. Rev. E 89, 042119 (2014)
Crutchfield, J.P., McNamara, B.S.: Equations of motion from a data series. Complex Syst. 1, 417–452 (1987)
Johnson, B.D., Crutchfield, J.P., Ellison, C.J. McTague, C.S.: Enumerating finitary processes. Santa Fe Institute Working Paper 10–11-027. arXiv:1011.0036 [cs.FL]
Shalizi, C.R., Shalizi, K.L., Crutchfield, J.P.: Pattern discovery in time series, Part I: Theory, algorithm, analysis, and convergence. Santa Fe Institute Working Paper 02–10-060. arXiv:cs.LG/0210025
Still, S., Crutchfield, J.P., Ellison, C.J.: Optimal causal inference: estimating stored information and approximating causal architecture. CHAOS 20(3), 037111 (2010)
Crutchfield, J.P.: Reconstructing language hierarchies. In: Atmanspracher, H.A., Scheingraber, H. (eds.) Information Dynamics, pp. 45–60. Plenum, New York (1991)
Hanson, J.E., Crutchfield, J.P.: The attractor-basin portrait of a cellular automaton. J. Stat. Phys. 66, 1415–1462 (1992)
Crutchfield, J.P.: Discovering coherent structures in nonlinear spatial systems. In: Brandt, A., Ramberg, S., Shlesinger, M. (eds.) Nonlinear Ocean Waves, pp. 190–216. World Scientific, Singapore (1992)
Hanson, J.E., Crutchfield, J.P.: Computational mechanics of cellular automata: An example. Physica D 103, 169–189 (1997)
McTague, C.S., Crutchfield, J.P.: Automated pattern discovery—an algorithm for constructing optimally synchronizing multi-regular language filters. Theoe. Comp. Sci. 359(1–3), 306–328 (2006)
Crutchfield, J.P., Görnerup, O.: Objects that make objects: the population dynamics of structural complexity. J. R. Soc. Interface 3, 345–349 (2006)
Crutchfield, J.P., Whalen, S.: Structural drift: the population dynamics of sequential learning. PLoS Comput. Biol. 8(6), e1002510 (2010)
J. P. Crutchfield. Optimal structural transformations-the \(\epsilon \)-transducer. UC Berkeley Physics Research Report, 1994
Shalizi, C.R.: Causal architecture, complexity and self-organization in time series and cellular automata. PhD thesis, University of Wisconsin, Madison (2001)
Hopfield, J.J.: Neural networks and physical systems with emergent collective behavior. Proc. Natl. Acad. Sci. 79, 2554 (1982)
Hertz, J., Krogh, A., Palmer, R.G.: An Introduction to the Theory of Neural Networks. Lecture Notes, vol. 1. Studies in the Sciences of Complexity. Addison-Wesley, Redwood City (1991)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Spinger, New York (2011)
MacKay, D.J.C.: Information Theory. Inference and Learning Algorithms. Cambridge University Press, Cambridge (2003)
Bialek, W., Nemenman, I., Tishby, N.: Predictability, complexity, and learning. Neural Comput 13, 2409–2463 (2001)
Shalizi, C.R., Crutchfield, J.P.: Information bottlenecks, causal states, and statistical relevance bases: How to represent relevant information in memoryless transduction. Adv. Comput. Syst. 5(1), 91–95 (2002)
Albers, D., Sprott, J.C., Crutchfield, J.P.: Persistent chaos in high dimensions. Phys. Rev. E 74(5), 057201 (2006)
Marzen, S., Crutchfield, J.P.: Informational and causal architecture of discrete-time renewal processes. Entropy 17(7), 4891–4917 (2015)
Crutchfield, J.P.: Information and its metric. In: Lam, L., Morris, H.C. (eds.) Nonlinear Structures in Physical Systems—Pattern Formation. Chaos and Waves, pp. 119–130. Springer, New York (1990)
Shannon, C.E.: The lattice theory of information. IEEE Trans. Inf. Theory 1, 105–107 (1953)
Li, H., Chong, E.K.P.: On a connection between information and group lattices. Entropy 13, 683–798 (2011)
Morse, M., Hedlund, G.A.: Symbolic dynamics. Am. J. Math. 60(4), 815–866 (1938)
Post, E.: Introduction to the general theory of elementary propositions. Am. J. Math. 43, 163–185 (1921)
Gödel, K.: On Formally Undecidable Propositions of Principia Mathematica and Related Systems. Dover Publications, Mineola (1992)
Turing, A.: On computable numbers, with an application to the Entschiedungsproblem. Proc. Lond. Math. Soc. 42, 43:230–265, 544–546 (1937)
Church, A.: A note on the Entscheidungsproblem. J. Symb. Log. 1, 40–41 (1936)
Moore, E.F.: Gedanken-experiments on sequential machines. In: Shannon, C., McCarthy, J. (eds.) Automata Studies. number 34 in Annals of Mathematical Studies, pp. 129–153. Princeton University Press, Princeton, New Jersey (1956)
Huffman, D.: The synthesis of sequential switching circuits. J. Frankl Inst. 257(161–190), 275–303 (1954)
Huffman, D.: Information conservation and sequence transducers. In: Proceedings of the Symposium on Information Networks, pp. 291–307. Polytechnic Institute of Brooklyn, Brooklyn (1954)
Huffman, D.: Canonical forms for information-lossless finite-state logical machines. IRE Trans. Circuit Theory 6, 41–59 (1959)
Huffman, D.: Notes on information-lossless finite-state automata. Il Nuovo Cimento 13(2 Supplement), 397–405 (1959)
Brookshear, J.G.: Theory of Computation: Formal Languages, Automata, and Complexity. Benjamin/Cummings, Redwood City (1989)
Ash, R.B.: Information Theory. Wiley, New York (1965)
Blackwell, D., Breiman, L., Thomasian, A.J.: Proof of Shannon’s transmission theorem for finite-state indecomposable channels. Ann. Math. Stat. 29(4), 1209–1220 (1958)
Blackwell, D.: Exponential error bounds for finite state channels. In: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 57–63. University of California Press, (1961)
Hamming, R.W.: Digital Filterns, 3rd edn. Dover Publications, New York (1997)
Bateson, G.: Mind and Nature: A Necessary Unity. E. P. Dutton, New York (1979)
R. Shaw. Strange attractors, chaotic behavior, and information flow. Z. Naturforsh., 36a:80, 1981
Ahlswede, R., Körner, J.: Appendix: on common information and related characteristics of correlated information sources. In: Ahlswede R., Baumer, Cai N., Aydinian H., Blinovsky V., Deppe C., Mashurian H. (eds.) General Theory of Information Transfer and Combinatorics. Lecture Notes in Computer Science, vol. 4123, pp. 664–677. Springer, Berlin (2006)
Williams, P.L., Beer, R.D.: Generalized measures of information transfer. arXiv:1102.1507
Barnett, L., Barrett, A.B., Seth, A.K.: Granger causality and transfer entropy are equivalent for Gaussian variables. Phys. Rev. Lett. 103, 238701 (2009)
Schreiber, T.: Measuring information transfer. Phys. Rev. Lett. 85, 461–464 (2000)
Marko, H.: The bidirectional communication theory—a generalization of information theory. IEEE Trans. Commun. 21, 1345–1351 (1973)
Massey, J.L.: Causality, feedback and directed information. In: Proceedings of the 1990 International Symposium on Information Theory and Its Applications, pp. 1–6. Waikiki, Hawaii, 27–30 Nov 1990
Sun, J., Bollt, E.M.: Causation entropy identifies indirect influences, dominance of neighbors and anticipatory couplings. Physica D 267, 49–57 (2014)
Barnett, N., Crutchfield, J.P.: Computational mechanics of input-output processes: Shannon information measures and decompositions. in preparation (2014)
Fedorov, V.V.: Theory of Optimal Experiments. Probability and Mathematical Statistics. Academic Press, New York (1972)
Atkinson, A., Bogacka, B., Zhigljavsky, A.A. (eds.): Optimum Design 2000. Nonconvex Optimization and Its Applications. Springer, New York (2001)
Still, S.: Information-theoretic approach to interactive learning. EuroPhys. Lett. 85, 28005 (2009)
Astrom, K.J., Murray, R.M.: Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press, Princeton (2008)
Crutchfield, J.P., Riechers, P., Ellison, C.J.: Exact complexity: spectral decomposition of intrinsic computation. Santa Fe Institute Working Paper 13–09-028. arXiv:1309.3792 [cond- mat.stat-mech]
Acknowledgments
We thank Cina Aghamohammadi, Alec Boyd, David Darmon, Chris Ellison, Ryan James, John Mahoney, and Paul Riechers for helpful comments and the Santa Fe Institute for its hospitality during visits. JPC is an SFI External Faculty member. This material is based upon work supported by, or in part by, the U. S. Army Research Laboratory and the U. S. Army Research Office under contract numbers W911NF-12-1-0234, W911NF-13-1-0390, and W911NF-13-1-0340. NB was partially supported by NSF VIGRE Grant DMS0636297.
Author information
Authors and Affiliations
Corresponding author
Appendix: Equivalence of Two \(\epsilon \)-Transducer Definitions
Appendix: Equivalence of Two \(\epsilon \)-Transducer Definitions
We show the equivalence of two different \(\epsilon {\text {-}}\mathrm transducer\) definitions, that presented in the main paper and an earlier version requiring additional assumptions. Since the \(\epsilon {\text {-}}\mathrm transducer\) is determined by its causal equivalence relation, we show that the respective equivalence relations are the same. The first is defined and discussed at length above and duplicated here for convenience.
Definition 1
The causal equivalence relation \(\sim _\epsilon \) for channels is defined as follows:
The second definition is an implicit equivalence relation consisting of an explicit equivalence relation, along with an additional unifilarity constraint that, of course, is quite strong [48, 49]. Here, we make both requirements explicit.
Definition 2
The single-symbol unifilar equivalence relation \(\sim _\epsilon ^1\) for channels is defined as follows:
for all \(a \in \varvec{\mathcal {X}}\) and \(b \in \varvec{ \mathcal {Y}}\) such that:
and:
The second requirement (ii) in the above definition requires that appending any joint symbol to two single-symbol-equivalent pasts will also result in a pair of pasts that are single-symbol-equivalent. This is unifiliarity. The second part of the second requirement ensures that we are only considering possible joint symbols (a, b)—symbols that can follow or with some nonzero probability.
Proposition 7
The single-symbol unifilar equivalence relation is identical to the causal equivalence relation.
Proof
Let and be two pasts, equivalent under \(\sim _\epsilon ^1\). This provides our base case for induction:
Now, let’s assume that and are equivalent for length-\(L-1\) future morphs:
We need to show that and are equivalent for length-L future morphs by using the unifilarity constraint. Unifilarity requires that appending a joint symbol (a, b) to both and results in two new pasts also equivalent to each other for length-\(L-1\) future morphs:
Since this must be true for any joint symbol, we replace (a, b) with \(( X, Y)_0\) in Eq. (15), giving:
To arrive at our result, we need to multiply the left side of Eq. (16) by and the right side by , which we can do when these quantities are equal. Since our channel is causal, \( X_{1:L+1}\) has no effect on \( Y_0\) when we condition on the infinite joint past and present input symbol. The two quantities, and , therefore reduce to and , respectively. But these are equal by the single-symbol unifilar equivalence relation—the base for induction. Multiplying each side of Eq. (16) by these two terms (in their original form) gives:
The two pasts are therefore equivalent for length-L future morphs. By induction, the two pasts are equivalent for arbitrarily long future morphs. \(\square \)
Rights and permissions
About this article
Cite this article
Barnett, N., Crutchfield, J.P. Computational Mechanics of Input–Output Processes: Structured Transformations and the \(\epsilon \)-Transducer. J Stat Phys 161, 404–451 (2015). https://doi.org/10.1007/s10955-015-1327-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10955-015-1327-5