Now, here, you see, it takes all the running you can do, to keep in the same place.

Lewis Carroll, Through the Looking-Glass

I remember well one day, as an undergraduate student, hearing someone talking about a theory of evolution that was somewhat inspired by a conversation between Alice and the Red Queen in Through the Looking-Glass (Carroll [1871] 1999). It was something about ecosystems as a whole as an intrinsic part of the “environment” of each species. I was already interested in complexity and the concept that the whole was more than the sum of the parts. I eventually found a copy of the 1973 typewritten article: “A New Evolutionary Law,” (Van Valen 1973) not elegantly edited as with most top journals but instead looking homemade (or as paleobiologist David Jablonski puts it, having an “incidental esthetics”).Footnote 1 The paper contained two parts, both fascinating.

The first dealt with evidence from fossil record data of a seemingly universal “law of constant extinction”: for every example studied (with exceptions, as always occur in biology) the members of a given phylogenetic group, from Foraminifera to Ammonoidea to reptiles and mammals, appeared to become extinct with essentially the same probability, as if the duration since their origin were not relevant at all. In the second part, Van Valen suggested that one way of interpreting the constant extinction could be a “Red Queen dynamics” scenario, where all species change while experiencing “mutually incompatible optima within an adaptive zone. A self-perpetuating fluctuation results which can be stated in terms of an unstudied aspect of zero-sum game theory.”Footnote 2 In other words, each species tries to improve its fitness, but in doing so it modifies the evolutionary responses of its partners in the ecological network which then also change, triggering further responses in the former.

This was in fact a higher-level narrative of the coevolutionary dynamics picture: a prey becomes more cryptic or faster, and its predator therefore needs to improve its sight or speed. Each of these changes occurs in a context where costs and trade-offs are present. Not all changes are possible or limitless. But when many species are involved, a much more complex situation arises, where each one responds to (and affects) the changes in the others, i.e., the “biotic environment.” The consequence would be a rather unexpected one: everyone needs to keep changing to remain in the (evolutionary) game. In that perspective, a reductionist view of ecosystems as sets of species whose fates would depend on environmental changes was replaced by a very different one with extinction being the result of intrinsic, system-level changes.

Fig. 1
figure 1

Van Valen and the rise of the Red Queen. The impact of Van Valen’s Red Queen hypothesis can be appreciated in the right panel, where the Google ngrams viewer tool (https://books.google.com/ngrams) has been used to measure the relative frequency of the term “Red Queen” within a very large corpus of written texts. The plot marks the publication of Lewis Carroll’s Through the Looking-Glass (drawing by R. Solé) and the publication of Van Valen’s classic paper where his theory was first introduced. A rapid growth in usage is noticeable, indicating a marked spread of the concept within the scientific literature

Full size image

I was not aware then that the ultimately enormously influential paper had been rejected by several leading journals, including Nature. That chain of rejections led to the creation, by Van Valen himself, of a new journal, Evolutionary Theory, the first article being none other than the constant extinction and the Red Queen hypothesis. Despite its difficult beginnings, the article started to become very well known, eventually generating a major response. Nowadays, it is considered by many to be one of the most influential papers in evolutionary biology of the 20th century, and its importance has not faded since. A visual picture of this success is provided by Fig. 1, where we display the time series of citations of the term “Red Queen” as found in a digital archive of millions of books. The software detects the number of occurrences per year, and two landmark dates are highlighted: the publication of Lewis Carroll’s book Through the Looking-Glass (the follow up to Alice’s Adventures in Wonderland) and the publication year of Van Valen’s paper. Although Carroll’s book was famous in 1973, we can see a marked increase of citations in the aftermath of “A New Evolutionary Law” that have to do with its scientific relevance.

The idea of a constantly changing, coevolving ecological theater spread fast and all over. It quickly became adopted and tested by evolutionary biologists and researchers in prey-predator or host-pathogen interactions, and eventually inspired a plethora of mathematical and computer models (Brockhurst et al. 2014). Scholars within economics and sociology used game theory to analyze the problem, and physicists developed new theories of changing fitness landscapes. The Red Queen was reigning in the kingdom of evolutionary biology and beyond.

The Constant Extinction Pattern

Van Valen observed that the vast majority of taxonomic groups analyzed displayed exponentially decaying survivorship curves. This result implied constancy in the probability of extinction of the taxa, regardless of their previous duration. That is, both data from the fossil record and from extant species suggested that a given species may disappear at any time, irrespective of how long it has already existed. This unexpected phenomenon (the Law of Constant Extinction), can be formulated in a simple way, as follows. If N(t) indicates the number of species at a given time, and we follow their presence over time (ignoring other events) we would observe an exponential decay law, namely:

$$\begin{aligned} \frac{dN}{dt} = - \delta (t) N \end{aligned}$$
(1)

where \(\delta (t)\) indicates a time-dependent extinction rate. If \(N_0\) is the original cohort size, this differential equation is easily solved, and gives:

$$\begin{aligned} N(t) = N_0 \exp \left( -\int _0^t \delta (t) dt\right) \end{aligned}$$
(2)

Despite the seemingly obvious assumption that \(\delta\) depends on t, the surprising observation is that the observed curves fit very well a constant decay rate \(\delta\), i.e., a solution:

$$\begin{aligned} N(t)=N_0 e^{-\delta t} \end{aligned}$$
(3)

where \(\delta\) is the extinction probability of a species (per millions of years, Myr). A median longevity time \(\tau\) can be estimated from this equation, defined as the time required to halve the initial cohort size, i.e., \(N(\tau )=N_0/2\). It is easy to show that this time is just

$$\begin{aligned} \tau = {Ln 2 \over \delta } \end{aligned}$$
(4)

which can be considered a law of paleobiology (Marshall 2017). The validity of this law was soon questioned and became a controversial issue (Liow et al. 2011). But for some well-preserved taxa, a constant extinction rate seems consistent with data (Benton 1987, 1995). In particular, dedicated studies revealed that species of modern mammals are just as likely to become extinct as were their ancestors living 200 Myr ago (Benton 1995). If evolution leads to improvement through adaptation, why do modern mammals have the same extinction probabilities as their ancestors? Enter the Red Queen.

Fig. 2
figure 2

Experimental systems displaying evolutionary arms races include, for example, the host-parasite system formed by Daphnia magna (a, photo courtesy of Ted Kinsman) and its pathogen (b) Pasteuria ramosa, here shown in two different stages in its life cycle (photo courtesy of Dieter Ebert)

Full size image

Never-Ending Coevolution

The fossil record offers a description of biological change that is rich and revealing: we know that around \(99 \%\) of all species that ever lived are now extinct (Raup 1986). Moreover, external events are known to play a role in driving large extinction events. In this context, the global pattern of decay of taxonomic cohorts is much more episodic, marked by sharp decays associated with five major extinction events (Raup 1986). But despite these abrupt, coherent events, the background extinction among major decays suggests a steady extinction. In Van Valen’s paper, the explanation for this background extinction is a highly dynamical but internal phenomenon involving the simultaneous coevolution of many species within communities. Coevolution pervades evolutionary change on multiple scales and it is no exaggeration to say, to paraphrase Dobzhansky’s statement, that nothing makes sense in biology except in the light of coevolution. Darwin himself recognized this when referring to what he called the entangled bank (Darwin 1859):

It is interesting to contemplate an entangled bank, clothed with many plants of many kinds, with birds singing on the bushes, with various insects flitting about, and with worms crawling through the damp earth.

Indeed, ecosystems need to be seen as collectives of interacting species whose evolutionary fate is necessarily intermingled in complex ways. Van Valen’s Red Queen hypothesis (RQH) suggests that the constant change of the nodes of these networks propagates through the web as other species also respond by evolving themselves. Since change might not happen indefinitely (due to biological constraints) a species unable to cope with change will disappear. Such a hypothesis was mathematically formulated by Stenseth and Maynard Smith by means of a simple, two-dimensional model representing two variables: the average lag of different species and the number of species (Stenseth and Maynard Smith 1984). These authors showed that a constantly changing regime can be possible under the absence of environmental stress. It was also the beginning of a new generation of models mixing both ecological and evolutionary dynamics that have been under development since then. But just as living entities evolve, so do the theories.

What is the relative importance of biotic versus abiotic components in driving evolution? This is of course the key question, and the final answer might be dependent on the scales at which we watch the system. Proving the theory from paleontological data is far from trivial. Research on Cenozoic terrestrial mammals (Quental and Marshall 2013) strongly supports the RQH over environmental-driven causes. This might also apply to other cases studies including dinosaur evolution (Benson et al. 2014), but in other studies, a mixture of both internal and environmental factors are revealed. What is then the role of each component? Anthony Barnosky suggested a unifying picture where both biotic and abiotic components are included (Barnosky 2001). Under this Court Jester model, “evolution, speciation, and extinction rarely happen except in response to unpredictable changes in the physical environment, recalling the capricious behavior of the licensed fool of Medieval times” (Benton 2009). This more general theory (not yet truly formalized) predicts that the eco-evolutionary factors defining the Red Queen dynamics would surely dominate evolution over short time periods, whereas multiple factors such as climate variability or tectonic dynamics would dominate larger scales both in time and space.

Fig. 3
figure 3

The multiple scales of ecological and evolutionary change. Here, different levels of complexity involve new phenomena that cannot be reduced to the previous scale. Here we show: a single species, b pairwise interactions, c ecological networks, and d evolving communities on a macro ecological timescale. Despite the dominant species-level picture of evolution grounded in gene frequencies, the addition of interactions (even pairwise) leads to novel phenomena. Each scale has been explored by means of a diverse range of mathematical models (e): from bottom to top, novel features are included such as species-species interactions, diversity, evolving interactions, or large-scale macroevolution. This increasing complexity inevitably leads to increasing unpredictability as network-level traits become dominant

Full size image

The possibility that the Red Queen reigns in the shorter timescales has actually been tested empirically (Liow et al. 2011; Brockhurst et al. 2014). Many different case studies confirm the presence of this dynamical regime. A wide range of both theoretical models and microcosm experiments as well as the study of intrahost dynamics of pathogens include, among others: (a) transfer experiments of RNA viruses infecting mammalian cells (Clarke et al. 1994; Solé et al. 1999), (b) coevolving pathogens and their hosts (Agrawal and Lively 2002; Jaenike 1978; Hamilton 1980; Ndung’u and Weiss 2012), or (c) in the coevolution of Daphnia magna (Fig. 2a) and its parasitic bacterium Pasteuria ramosa (Fig. 2b) Decaestecker et al. (2007); Ebert (2008). The latter is an especially nice example that allowed proof of the consistency of the observed host-parasite dynamics with negative frequency-dependent selection, as predicted by the RQH.

How are the Red Queen dynamics and the constant extinction trend related? In Van Valen’s picture, they are two sides of the same coin. Because of the intrinsic complexity of ecological interactions, community-level changes will sooner or later create negative effects on any species that will jeopardize survival. Moreover, as species disappear, ecosystem rearrangements will also change the interaction balances. This cannot be reduced to our understanding of single-species features. Instead, it is a systems-level phenomenon. To make sense of it, a network approach is required.

Evolving Networks

An undeniable outcome of the RQH has been its inspiration for theoretical developments on coevolutionary dynamics and in particular with the use of fitness landscapes as a natural way of coupling coevolutionary interactions (Solé and Sardanyés 2014). The range of models spans multiple scales (Fig. 3) from pairwise interacting systems to macroevolutionary dynamics. With the rise of network science since the early 2000s, the old systems tradition of ecology (particularly within theoretical ecology) has been enriched with the use of both graph theory tools and statistical physics approaches (Drossel 2001; Newman and Palmer 2003). In particular, models of large-scale evolution and extinction have shown that the law of constant extinction might be a byproduct of a tendency of coevolving networks to move toward an unstable state (Solé et al. 1996).

The diverse theoretical developments of Van Valen’s picture have flourished in many directions, including (a) the early game theoretic models (Stenseth and Maynard Smith 1984), (b) predator-prey and host-parasite evolution (Ikegami and Kaneko 1992; Kaneko and Ikegami 1992; Van Der Laan and Hogeweg 1995; Morran et al. 2011; Rabajante et al. 2015), (c) microbiota dynamics (Bonachela et al. 2017), (d) the evolution of plant shapes in a virtual environment (Ebner 2006; Fernandez et al. 2012), (e) coevolution of robots (Nolfi and Floreano 1998), and (f) dynamical complexity in economics (Barnett and Hansen 1996; Robson 2005). It also inspired a new generation of statistical physics models of evolution (Drossel 2001; Newman and Palmer 2003). One of the main triggers for the latter is to be found in a landmark paper by Stuart Kauffman and Sönke Johnsen on “Coevolution at the Edge of Chaos” Kauffman and Johnsen (1991); Bak et al. (1992). The model presented by these authors proposed a well-defined framework to describe Red Queen dynamics in terms of coupled species equipped with a simple “genome” represented as strings of bits. Each of these genomes has an associated fitness measure, and mutations push each species up within their fitness landscapes. However, instead of the original conception of fitness landscape as a species-specific structure, the landscapes of different species are coupled in such a way that a change in a given landscape can propagate into the others. Under this general framework, Red Queen dynamics as well as punctuated equilibrium (Eldredge and Gould 1972) would be an inevitable result of a tendency to critical states (Bak et al. 1992; Gould 2002) where ecosystem-level coevolution will lead to unpredictable dynamics, effectively decoupling micro- from macroevolution (Solé et al. 1996).

The Red Queen has inspired further evolutionary metaphors, including (a) the Red King dynamics of mutualistic communities, where the slowly evolving species is likely to gain a disproportionate share of the benefits (instead of the faster changing one) (Bergstrom and Lachmann 2003), (b) the Black Queen hypothesis, which proposes that gene loss can provide a selective advantage by conserving an organism’s limiting resources (Morris et al. 2012), and especially relevant for microbial communities (c) the Suicide King, a phenomenon related to the maladaptive coevolution of parasites that can lead to the extinction of their hosts (Dybdahl and Storfer 2003).

Van Valen had a wide range of interests, spanning the history of all life forms. He had a mind for “out of the box” ideas, such as the concept of cancers as biological species, exemplified by the cancerous HeLa cell line (Van Valen 1991), which has inspired others to reconsider the nature of cancer populations under a different view (Duesberg and Rasnick 2000). “I don’t work linearly,” he explained in a note to his department chair. “I am a generalist and tend to open new approaches more than fill them in. What I work on changes irregularly and unpredictably with the progress of theory and knowledge” (UChicago News 2010). His RQH was certainly a nonlinear, extraordinary insight. The key concept has been shown to be correct and central to our understanding of evolution (de Vladar et al. 2017). In many ways, Van Valen can be said to have been able to peer through the Looking Glass and foresee a whole land of open-ended possibilities.