Three faces of entropy for complex systems: Information, thermodynamics, and the maximum entropy principle

Stefan Thurner, Bernat Corominas-Murtra, and Rudolf Hanel
Phys. Rev. E 96, 032124 – Published 15 September 2017

Abstract

There are at least three distinct ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a means for statistical inference on multinomial processes (Jaynes maximum entropy principle). Even though these notions represent fundamentally different concepts, the functional form of the entropy for thermodynamic systems in equilibrium, for ergodic sources in information theory, and for independent sampling processes in statistical systems, is degenerate, H(p)=ipilogpi. For many complex systems, which are typically history-dependent, nonergodic, and nonmultinomial, this is no longer the case. Here we show that for such processes, the three entropy concepts lead to different functional forms of entropy, which we will refer to as SEXT for extensive entropy, SIT for the source information rate in information theory, and SMEP for the entropy functional that appears in the so-called maximum entropy principle, which characterizes the most likely observable distribution functions of a system. We explicitly compute these three entropy functionals for three concrete examples: for Pólya urn processes, which are simple self-reinforcing processes, for sample-space-reducing (SSR) processes, which are simple history dependent processes that are associated with power-law statistics, and finally for multinomial mixture processes.

  • Figure
  • Figure
  • Received 6 May 2017
  • Revised 4 August 2017

DOI:https://doi.org/10.1103/PhysRevE.96.032124

©2017 American Physical Society

Physics Subject Headings (PhySH)

Statistical Physics & Thermodynamics

Authors & Affiliations

Stefan Thurner1,2,3,4, Bernat Corominas-Murtra1,4, and Rudolf Hanel1,4

  • 1Section for the Science of Complex Systems, CeMSIIS, Medical University of Vienna, Spitalgasse 23, A-1090 Vienna, Austria
  • 2Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, New Mexico 87501, USA
  • 3IIASA, Schlossplatz 1, 2361 Laxenburg, Austria
  • 4Complexity Science Hub Vienna, Josefstädterstrasse 39, A-1090 Vienna, Austria

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand
Issue

Vol. 96, Iss. 3 — September 2017

Reuse & Permissions
Access Options
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review E

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×