^{Cite as: Hurt, Wyatt A., Alicia G. Harley, and William C. Clark. 2020. “Supplemental Materials: Research Programs That Have Shaped Sustainability Science.” In Sustainability Science: A Guide for Researchers, edited by Alicia G. Harley and William C. Clark, 1st ed. Retrieved from https://www.sustainabilityscience.org/pub/kzs257kc}

The main purpose of this Supplementary Material is to explain how we arrived at the list of influential research programs listed in Table 1 of this Research Guide (see An Integrative Framework for Sustainability Science). In addition, we provide additional information about those programs: their substantive focus, their “footprint” on sustainability science, and how future contributions from each of them can be tracked by interested researchers. Readers interested in a deeper dive into the bibliometric analyses that underlie the information summarized here are encouraged to consult a longer working paper we have prepared (Hurt, Harley, and Clark 2020). Wyatt Hurt did much of the data collection and analysis for this SI, and is the lead author on the working paper.

We derived the final set of influential research programs provided in Table 1 of the Research Guide from a longer preliminary list of candidate research programs that emerged from our initial discussions with colleagues, existing surveys of the sustainability science literature, and searches in various databases. Two big findings emerged from that preliminary work. First, the argument of Bettencourt and Kaur (2011) that the field of sustainability science had “unified” around the year 2000 may have been somewhat premature: many of the research programs in our candidate list are still relatively isolated from one another and are only now beginning to collaborate and draw from one another’s insights. Second, much scholarship that experienced researchers see as being highly relevant to sustainable development does not describe itself as “sustainability science.” That work is missed by literature searches requiring only that some variant of the word “sustainable” appear in titles, abstracts, or key words. Most published bibliometric studies seeking to identify and analyze scholarly contributions to our understanding of sustainable development have adopted just such expedients and are therefore seriously incomplete (e.g., Bettencourt and Kaur 2011; Buter and Van Raan 2013; Kajikawa, Saito, and Takeuchi 2017; Fang et al. 2018).

The strategy we found most useful to address the sobering findings of our preliminary survey of research on sustainability may be summarized as follows:

• We began by treating each candidate research program as a potential source of scholarship relevant to sustainability, regardless of whether its publications commonly used the term.
  

• Next, we sought to design a set of search terms that would reliably identify the subset of works published in that program that qualified as “sustainability science” as we had defined the field in our Research Guide.1 These search terms, and our strategy more generally, could in principle be applied to any database, including the “everything (books+journals)” catalogs of holdings in particular libraries now being offered by an increasing number of universities.2 As a practical matter, we applied our terms only to the Web of Science (WoS), with all the advantages and limitations that use of this particular database entails.3 In particular, we searched all indexes in the Web of Science Core Collection, restricting results to English-language articles, books, and book chapters only. (Note that the Research Guide itself cites works from a larger range of sources, not all indexed by WoS.)

• Our tuning of the search terms depended heavily on our own determination of whether the works a given combination of terms identified should qualify as “sustainability science.” We made the “qualification” judgment through our own (admittedly subjective) reading of the title, abstract, and often substantial fractions of the body of the text in question. We performed this judgment for the most recent 100 works identified by a given configuration of search terms.4 Most of the works that we “disqualified” through this close reading were clearly irrelevant to sustainable development. In addition, however, we disqualified a non-negligible number of works because they fell outside of the Pasteur’s Quadrant focus of our Research Guide, i.e., they were plausibly relevant to sustainability, but were too much Bohr’s Quadrant (basic research that may have provide foundations for sustainability studies but did not explicitly build sustainability arguments from those foundations) or too much Edison’s Quadrant (applications to particular situations with little to offer our search for generalizable knowledge).
  

• We continued tuning the search terms until we achieved an acceptable balance of Type 1 and Type 2 errors. Our first step was to achieve a result in which Type 1 errors (works identified by the search that we ultimately disqualified) were less than 10% of all the items identified in the search. (Most of our initial search term combinations in fact yielded a Type 1 error rate of less than 5%.) To limit Type 2 errors (works that we judged to be qualified, and thus should have been identified by the search, but were not), we experimented with less restrictive variations of each search. Our goal was to drive the Type 2 error rate down while maintaining a Type 1 error rate of 10% or less. We refer in what follows to the resulting list as “qualified works,” meaning that—in our judgment—they qualify as works centered in Pasteur’s Quadrant and relevant to sustainable development. In short, they qualified as the “sustainability science” that was the focus of the Research Guide.

The results of applying our strategy are summarized in Table SM1. Here are some highlights:

• What search terms are needed to identify the most important works in sustainability science (column B)? We ultimately designed a search query combination that achieved our desired Type 1/Type 2 balance for each individual research program we investigated. These search query combinations are provided in Column B of Table SM1. This methodology is imperfect. Nevertheless, we think it is a powerful and comprehensive approach to understanding the broad contours of research contributing to sustainability science. Our results confirm the sad fact we noted earlier: any search strategy based on variations of the word “sustainable” alone will both turn up a lot of false positives (e.g., “sustainable sales strategy”) and miss a significant fraction of the all the scholarship relevant to sustainable development. Indeed, about a third of the total set of qualified works returned by the search queries in Table SM1 would have been missed by “sustainability” searches prevalent among bibliometric analyses of sustainability science, underscoring the need for more complex search strategies. To be practical and effective these search strategies must be tailored to the individual research programs listed in Table SM1. We have found it useful to create weekly WoS alerts for each of these search queries as a relatively easy if imperfect way of having our attention called to new publications in each research program on a regular basis. Here is how: https://clarivate.libguides.com/woscc/saving. We hope our readers will find the search queries useful for similar or related purposes of their own. We welcome suggestions for how the search queries can be used, and improved, to support the pursuit of sustainability.

• How broad is the emerging field of sustainability science (Column C)? The magnitude of each research program’s offerings to sustainability science (specifically, to the Pasteur’s Quadrant part of sustainability science) is shown in Column C: the count of qualified works in WoS identified by the search string in Column B for the period 2000–2019. Note that research programs that were included in our preliminary list, but that achieved very low scores on this metric, are excluded from this table and from the simpler version of it published in the Research Guide. In particular, new research programs that are just getting started may not yet score high enough to be included. Beyond these caveats, for all the reasons noted above, not much weight should be placed on the specific or perhaps even the relative counts listed in Column C: there are many limitations to the WoS database and different search queries would have produced different counts. Nonetheless, we were surprised by some of the results. And we suspect that most readers will find, as we did, that whichever research program(s) they consider to be their own “home” or starting place, there exist other programs that have produced a larger number of relevant offerings and therefore perhaps merit more attention than we are giving them in our respective efforts to shape a synoptic perspective on the emerging field of sustainability science.
  

• Trends in number of qualified works in sustainability science (Column D): Trends in the number of qualified sustainability science works for each research program are shown in Column D. (The time period covered by our searches was usually set to cover only the period 2000–2019 because this Research Guide was meant to cover that period, not the broader history of scholarship relevant to sustainable development. For this column only we extended the time period back to 1980 to let us explore which programs started early and which started late.) The general surge in publications over the last decade is clear if not surprising: the overall scientific literature relevant to sustainable development is now doubling every 3–4 years. (Column D shades in blue for each research program the years during which the most recent 50% of all qualified works were published. For the qualified works from all research programs combined—i.e., for what might be called the field of sustainability science—the time trend is shown in Table SM4.) More interesting, perhaps, is the suggestion from the trends of the listed research programs that some have long contributed to sustainability science and continue to do so today, while others are relatively new arrivals and still others may have already peaked in their contributions. We conclude that the field of sustainability science is growing rapidly in numbers of publications but also changing rapidly in terms of which research programs contribute most to those publications.

• How many journals must a scholar read to keep up with sustainability science (Column E)? The 5 journals that publish the greatest number of qualified works for each research program are listed in Column E. (Where there is a tie in fifth place, we list more than five journals). Note that publications in the research programs are spread across a very large number of journals: the most commonly used journal for each program rarely publishes as much of 10% of all its qualified works. For all research programs combined, only one journal appears in the list of “5 journals with the most qualified publications” for two thirds of the research programs, and only four journals appear for more than a one third (see Table SM2). The situation is only slightly less daunting when we turn to the list of journals cited in the Research Guide (see Table SM5). (These publications report what we judged to be some of the most significant and high-quality work in sustainability science. But to catch just one third of the articles cited in the Research Guide requires keeping up with 5 different journals; to catch one half requires 10 journals; to catch two thirds requires 20 different journals. We conclude that there is no “short list” of journals that will let a diligent reader keep up with the field of sustainability science. This stands on contrast to many scholarly fields in which where keeping up with two to three journals is sufficient to have a good grasp of the most important recent literature in the field.
  

• How many organizations must a scholar follow to keep up with sustainability science (Column F)? The 5 organizations whose researchers publish the greatest number of qualified works for each research program are listed in column F of Table SM1. Note that this research relevant to sustainable development is spread across a very large number of organizations, most of which specialize in only a few of the research programs that have contributed to the field. In particular, the most prolific organization houses researchers who publish only 3% of the papers assembled through our search queries across all research programs (Table SM3). And any one organization is a major player in at most half of the research programs listed in Table SM1. We conclude that there is no “short list” of organizations that dominate research production across the multiple research programs contributing to sustainability science.

• How influential are the various research programs that contribute to sustainability science (Column G)? The findings noted above have been based primarily on the total quantity of research output that our search strategies have identified as “qualifying” as, and contributing to, sustainability science. What we would most like to know, however, is the quality or influence of those programs. Col. G in Table SM1 reports a first step toward assessment of influence other than quantity. It does so by tabulating the number of citations to all of the works identified through our search queries in each of the listed research programs (Column C). The results, sadly, tell us little that we didn’t already know from the raw publication counts of Col. H: research programs that publish most get cited most. We conclude that there is no “short list” of research programs that dominate in the production of cited (and therefore, presumably, quality) research.

• Contributions to this Research Guide (columns H and I): Columns H and I of Table SM1 are the bridge from the details of bibliometric analysis reported here to the Research Guide itself. Drawing on our close reading of the literature for each research program, we summarize in Col. H the contributions particular to each program that have, in our subjective judgment, done most to influence the course of contemporary sustainability science. As we note in the body of the Research Guide, all the research programs listed in Table SM1 have been converging in recent years. They increasingly share the synoptic view of key questions, elements, and relationships that we have highlighted in the Research Guide. Historically, however, each of the research programs listed here has brought something special to the party. It is those special contributions that we note in Col. H. Column I indicates the review paper(s) that we believe provides the best recent overview of each listed research program.

Table SM1: Research Program Bibliometrics: Publications relevant to sustainability science indexed in the Web of Science

If you would like to annotate or comment on this table, please click here.

If you would like a printer-friendly pdf version of Table SM1, please click here.

Table SM2: Which journals publish the greatest number of works relevant to sustainability science (across all research programs)

This table displays: (1) journal name; (2) the total contribution of each journal as the percent of all qualified works identified in all of our searches (i.e., summed across all research programs, N = 20,813 works); (3) the number (N) of research programs for which each journal is one of those in which authors most frequently publish (i.e. number of appearances in Col. E of Table SM1).

Table SM3: Which organizations support authors responsible for the greatest number of qualified works (across all research programs)?

This table displays: (1) organization name; (2) the total contribution of each organization as the percent of works (articles, books, and book chapters) identified in our searches (i.e., summed across all research programs, N = 20,813 works); (3) the number (N) of research programs for which each organization is among those hosting researchers responsible for the greatest number of qualified works (i.e. number of appearances in Col. F of Table SM1.

Table SM4: Growth in Total Annual Production of Qualified Works

This plot displays the number of qualified works published each year, summed across all research programs listed in Table SM1 (period covered: 1990 (year 0) to 2019 (year 29)). Columns are shaded blue for the years during which the most recent 50% of all qualified works were published.

Table SM5: Journals Cited in the Research Guide (Clark & Harley 2020)

This table displays: (1) all journals cited by Clark & Harley 2020; (2) the count of times articles in those journals were cited in the Research Guide. The appearance of multiple Annual Review publications in the list reflects in part Clark and Harley’s effort to cite recent reviews of specialized topics relevant to sustainability science. We note that one of the authors of this SM (WCC) is a member of the editorial board of PNAS. We worked hard to assure that this did not bias our selection of which articles to cite in the Guide.

References

Bettencourt, Luís M. A., and Jasleen Kaur. 2011. “Evolution and Structure of Sustainability Science.” Proceedings of the National Academy of Sciences 108 (49): 19540–45. https://doi.org/10.1073/pnas.1102712108.

Buter, R. K., and A. F. J. Van Raan. 2013. “Identification and Analysis of the Highly Cited Knowledge Base of Sustainability Science.” Sustainability Science 8 (2): 253–67. https://doi.org/10.1007/s11625-012-0185-1.

Clark, William C., and Alicia G. Harley. 2020. “Sustainability Science: Toward a Synthesis.” Annual Review of Environment and Resources 45 (accepted preprint). http://nrs.harvard.edu/urn-3:HUL.InstRepos:42660129.

Fang, Xuening, Bingbing Zhou, Xingyue Tu, Qun Ma, and Jianguo Wu. 2018. “‘What Kind of a Science Is Sustainability Science?’ An Evidence-Based Reexamination.” Sustainability 10 (5): 1478. https://doi.org/10.3390/su10051478.

Garfield, Eugene. 1955. “Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas.” Science 122 (3159): 108–11. https://doi.org/10.1126/science.122.3159.108.

Gusenbauer, Michael. 2019. “Google Scholar to Overshadow Them All? Comparing the Sizes of 12 Academic Search Engines and Bibliographic Databases.” Scientometrics 118 (1): 177–214. https://doi.org/10.1007/s11192-018-2958-5.

Hurt, Wyatt W., Alicia G. Harley, and William C. Clark. 2020. “Searching for Sustainability Science: Practical Bibliometric and Curatorial Approaches.” 2020–02. Sustainability Science Program Working Paper. Cambridge, Massachusetts: John F. Kennedy School of Government, Harvard University.

Kajikawa, Yuya, Osamu Saito, and Kazuhiko Takeuchi. 2017. “Academic Landscape of 10 Years of Sustainability Science.” Sustainability Science 12 (6): 869–73. https://doi.org/10.1007/s11625-017-0477-6.

Stokes, Donald E. 1997. Pasteur’s Quadrant: Basic Science and Technological Innovation. Washington, DC: Brookings Institution Press.