Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Improving quantitative synthesis to achieve generality in ecology

Abstract

Synthesis of primary ecological data is often assumed to achieve a notion of ‘generality’, through the quantification of overall effect sizes and consistency among studies, and has become a dominant research approach in ecology. Unfortunately, ecologists rarely define either the generality of their findings, their estimand (the target of estimation) or the population of interest. Given that generality is fundamental to science, and the urgent need for scientific understanding to curb global scale ecological breakdown, loose usage of the term ‘generality’ is problematic. In other disciplines, generality is defined as comprising both generalizability—extending an inference about an estimand from the sample to the population—and transferability—the validity of estimand predictions in a different sampling unit or population. We review current practice in ecological synthesis and demonstrate that, when researchers fail to define the assumptions underpinning generalizations and transfers of effect sizes, generality often misses its target. We provide guidance for communicating nuanced inferences and maximizing the impact of syntheses both within and beyond academia. We propose pathways to generality applicable to ecological syntheses, including the development of quantitative and qualitative criteria with which to license the transfer of estimands from both primary and synthetic studies.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Generality, which we use synonymously with external validity, comprises the generalizability and transferability of estimands drawn from primary and synthetic research.
Fig. 2: Transferring an estimand to a target context.

Similar content being viewed by others

References

  1. Houlahan, J. E., McKinney, S. T., Anderson, T. M. & McGill, B. J. The priority of prediction in ecological understanding. Oikos 126, 1–7 (2017).

    Article  Google Scholar 

  2. Lawton, J. H. Are there general laws in ecology? Oikos 84, 177–192 (1999).

    Article  Google Scholar 

  3. Elliott-Graves, A. Generality and causal interdependence in ecology. Philos. Sci. 85, 1102–1114 (2018).

    Article  Google Scholar 

  4. Fox, J. W. The many roads to generality in ecology. Philos. Top. 9, 83–104 (2019).

    Article  Google Scholar 

  5. McGill, B. J. et al. Species abundance distributions: moving beyond single prediction theories to integration within an ecological framework. Ecol. Lett. 10, 995–1015 (2007).

    Article  PubMed  Google Scholar 

  6. MacArthur, R. H. & Wilson, E. O. An equilibrium theory of insular zoogeography. Evolution 17, 373–387 (1963).

    Article  Google Scholar 

  7. Gurevitch, J., Fox, G. A., Wardle, G. M., Inderjit & Taub, D. Emergent insights from the synthesis of conceptual frameworks for biological invasions. Ecol. Lett. 14, 407–418 (2011).

    Article  CAS  PubMed  Google Scholar 

  8. Borer, E. T. et al. Finding generality in ecology: a model for globally distributed experiments. Methods Ecol. Evol. 5, 65–73 (2014).

    Article  Google Scholar 

  9. Gurevitch, J., Koricheva, J., Nakagawa, S. & Stewart, G. Meta-analysis and the science of research synthesis. Nature 555, 175–182 (2018).

    Article  CAS  PubMed  Google Scholar 

  10. Anderson, S. C. et al. Trends in ecology and conservation over eight decades. Front. Ecol. Environ. 19, 274–282 (2021).

    Article  Google Scholar 

  11. Kneale, D., Thomas, J., O’Mara-Eves, A. & Wiggins, R. How can additional secondary data analysis of observational data enhance the generalisability of meta-analytic evidence for local public health decision making? Res. Synth. Methods 10, 44–56 (2019).

    Article  PubMed  Google Scholar 

  12. Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R. & Dalton, C. M. Debunking myths and urban legends about meta-analysis. Organ. Res. Methods 14, 306–331 (2011).

    Article  Google Scholar 

  13. Polit, D. F. & Beck, C. T. Generalization in quantitative and qualitative research: myths and strategies. Int. J. Nurs. Stud. 47, 1451–1458 (2010).

    Article  PubMed  Google Scholar 

  14. Cardinale, B. J., Gonzalez, A., Allington, G. R. H. & Loreau, M. Is local biodiversity declining or not? A summary of the debate over analysis of species richness time trends. Biol. Conserv. 219, 175–183 (2018).

    Article  Google Scholar 

  15. Lundberg, I., Johnson, R. & Stewart, B. M. What is your estimand? Defining the target quantity connects statistical evidence to theory. Am. Sociol. Rev. 86, 532–565 (2021).

    Article  Google Scholar 

  16. Lawrance, R. et al. What is an estimand & how does it relate to quantifying the effect of treatment on patient-reported quality of life outcomes in clinical trials? J. Patient-Rep. Outcomes 4, 68 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  17. Findley, M. G., Kikuta, K. & Denly, M. External validity. Annu. Rev. Polit. Sci. 24, 365–393 (2021).

    Article  Google Scholar 

  18. Pearl, J. & Bareinboim, E. External validity: from do-calculus to transportability across populations. Stat. Sci. 29, 579–595 (2014).

    Article  Google Scholar 

  19. Westreich, D., Edwards, J. K., Lesko, C. R., Cole, S. R. & Stuart, E. A. Target validity and the hierarchy of study designs. Am. J. Epidemiol. 188, 438–443 (2019).

    Article  PubMed  Google Scholar 

  20. Carpenter, C. J. Meta-analyzing apples and oranges: how to make applesauce instead of fruit salad. Hum. Commun. Res. 46, 322–333 (2020).

    Article  Google Scholar 

  21. Rohrer, J. M. & Arslan, R. C. Precise answers to vague questions: issues with interactions. Adv. Methods Pract. Psychol. Sci. 4, 1–19 (2021).

    Google Scholar 

  22. Breslow, N. E. & Clayton, D. G. Approximate inference in generalized linear mixed models. J. Am. Stat. Assoc. 88, 9–25 (1993).

    Google Scholar 

  23. Koricheva, J. & Gurevitch, J. Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102, 828–844 (2014).

    Article  Google Scholar 

  24. Gonzalez, A. et al. Estimating local biodiversity change: a critique of papers claiming no net loss of local diversity. Ecology 97, 1949–1960 (2016).

    Article  PubMed  Google Scholar 

  25. Konno, K. et al. Ignoring non-English-language studies may bias ecological meta-analyses. Ecol. Evol. 10, 6373–6384 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Nakagawa, S. et al. Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods Ecol. Evol. 13, 4–21 (2022).

    Article  Google Scholar 

  27. Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638–641 (1979).

    Article  Google Scholar 

  28. Leung, B. et al. Clustered versus catastrophic global vertebrate declines. Nature 588, 267–271 (2020).

    Article  CAS  PubMed  Google Scholar 

  29. Rothman, K. J., Gallacher, J. E. J. & Hatch, E. E. Why representativeness should be avoided. Int. J. Epidemiol. 42, 1012–1014 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  30. Spake, R. et al. Implications of scale dependence for cross-study syntheses of biodiversity differences. Ecol. Lett. 24, 374–390 (2021).

    Article  PubMed  Google Scholar 

  31. Spake, R. & Doncaster, C. P. Use of meta-analysis in forest biodiversity research: key challenges and considerations. For. Ecol. Manag. 400, 429–437 (2017).

    Article  Google Scholar 

  32. Christie, A. P. et al. Simple study designs in ecology produce inaccurate estimates of biodiversity responses. J. Appl. Ecol. 56, 2742–2754 (2019).

    Article  Google Scholar 

  33. Nakagawa, S., Noble, D. W. A., Senior, A. M. & Lagisz, M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 15, 18 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Higgins, J. P. T. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21, 1539–1558 (2002).

    Article  PubMed  Google Scholar 

  35. Schielzeth, H. & Nakagawa, S. Conditional repeatability and the variance explained by reaction norm variation in random slope models. Methods Ecol. Evol. 13, 1214–1223 (2022).

    Article  Google Scholar 

  36. Nakagawa, S. et al. The orchard plot: cultivating a forest plot for use in ecology, evolution, and beyond. Res. Synth. Methods 12, 4–12 (2021).

    Article  PubMed  Google Scholar 

  37. Lorah, J. Effect size measures for multilevel models: definition, interpretation, and TIMSS example. Large-Scale Assess. Educ. 6, 8 (2018).

    Article  Google Scholar 

  38. O’Connor, M. I. et al. A general biodiversity–function relationship is mediated by trophic level. Oikos 126, 18–31 (2017).

    Article  Google Scholar 

  39. Ojha, M., Naidu, D. G. T. & Bagchi, S. Meta-analysis of induced anti-herbivore defence traits in plants from 647 manipulative experiments with natural and simulated herbivory. J. Ecol. 110, 799–816 (2022).

  40. Dodds, K. C. et al. Material type influences the abundance but not richness of colonising organisms on marine structures. J. Environ. Manag. 307, 114549 (2022).

    Article  Google Scholar 

  41. Dornelas, M. et al. Assemblage time series reveal biodiversity change but not systematic loss. Science 344, 296–299 (2014).

    Article  CAS  PubMed  Google Scholar 

  42. Senior, A. M. et al. Heterogeneity in ecological and evolutionary meta- analyses: its magnitude and implications. Ecology 97, 3293–3299 (2016).

    Article  PubMed  Google Scholar 

  43. Blowes, S. A. et al. The geography of biodiversity change in marine and terrestrial assemblages. Science 366, 339–345 (2019).

    Article  CAS  PubMed  Google Scholar 

  44. Nakagawa, S. & Cuthill, I. C. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol. Rev. 82, 591–605 (2007).

    Article  PubMed  Google Scholar 

  45. Glass, G. V. Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3–8 (1976).

    Article  Google Scholar 

  46. Glass, G. V. Meta‐analysis at 25: a personal history. Education in Two Worlds https://ed2worlds.blogspot.com/2022/07/meta-analysis-at-25-personal-history.html (2000).

  47. Cooper, H. M. Organizing knowledge syntheses: a taxonomy of literature reviews. Knowl. Soc. 1, 104–126 (1988).

    Google Scholar 

  48. Soranno, P. A. et al. Cross-scale interactions: quantifying multi-scaled cause-effect relationships in macrosystems. Front. Ecol. Environ. 12, 65–73 (2014).

    Article  Google Scholar 

  49. Gerstner, K. et al. Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting. Methods Ecol. Evol. 8, 777–784 (2017).

    Article  Google Scholar 

  50. Hortal, J. et al. Seven shortfalls that beset large-scale knowledge of biodiversity. Annu. Rev. Ecol. Evol. Syst. 46, 523–549 (2015).

    Article  Google Scholar 

  51. Simons, D. J., Shoda, Y. & Lindsay, D. S. Constraints on Generality (CoG): a proposed addition to all empirical papers. Perspect. Psychol. Sci. 12, 1123–1128 (2017).

    Article  PubMed  Google Scholar 

  52. Yarkoni, T. The generalizability crisis. Behav. Brain Sci. https://doi.org/10.1017/S0140525X20001685 (2020).

  53. Lopez, P. M., Subramanian, S. V. & Schooling, C. M. Effect measure modification conceptualized using selection diagrams as mediation by mechanisms of varying population-level relevance. J. Clin. Epidemiol. 113, 123–128 (2019).

    Article  PubMed  Google Scholar 

  54. Campbell, D. T. in Advances in QuasiExperimental Design and Analysis (ed. Trochim, W.) 67–77 (Jossey-Bass, 1986).

  55. Spake, R. et al. Meta‐analysis of management effects on biodiversity in plantation and secondary forests of Japan. Conserv. Sci. Pract. 1, e14 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  56. Forest Ecosystem Diversity Basic Survey (in Japanese) (Forestry Agency of Japan, 2019); https://www.rinya.maff.go.jp/j/keikaku/tayouseichousa/index.html

  57. Ito, S., Ishigamia, S., Mizoue, N. & Buckley, G. P. Maintaining plant species composition and diversity of understory vegetation under strip-clearcutting forestry in conifer plantations in Kyushu, southern Japan. For. Ecol. Manag. 231, 234–241 (2006).

    Article  Google Scholar 

  58. Utsugi, E. et al. Hardwood recruitment into conifer plantations in Japan: effects of thinning and distance from neighboring hardwood forests. For. Ecol. Manag. 237, 15–28 (2006).

    Article  Google Scholar 

  59. Kominami, Y. et al. Classification of bird-dispersed plants by fruiting phenology, fruit size, and growth form in a primary lucidophyllous forest: an analysis, with implications for the conservation of fruit–bird interactions. Ornthological Sci. 2, 3–23 (2003).

    Article  Google Scholar 

  60. Tsujino, R. & Matsui, K. Forest regeneration inhibition in a mixed broadleaf-conifer forest under sika deer pressure. J. For. Res. 27, 230–235 (2021).

    Article  Google Scholar 

  61. Spake, R., Soga, M., Catford, J. A. & Eigenbrod, F. Applying the stress-gradient hypothesis to curb the spread of invasive bamboo. J. Appl. Ecol. 58, 1993–2003 (2021).

    Article  Google Scholar 

  62. Mize, T. D. Best practices for estimating, interpreting, and presenting nonlinear interaction effects. Sociol. Sci. 6, 81–117 (2019).

    Article  Google Scholar 

  63. Karaca-Mandic, P., Norton, E. C. & Dowd, B. Interaction terms in nonlinear models. Health Serv. Res. 47, 255–274 (2012).

    Article  PubMed  Google Scholar 

  64. Spake, R. et al. Forest damage by deer depends on cross-scale interactions between climate, deer density and landscape structure. J. Appl. Ecol. 57, 1376–1390 (2020).

  65. McCabe, C. J., Kim, D. S. & King, K. M. Improving present practices in the visual display of interactions. Adv. Methods Pract. Psychol. Sci. 1, 147–165 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  66. Shackelford, G. E. et al. Dynamic meta-analysis: a method of using global evidence for local decision making. BMC Biol. 19, 33 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  67. Christie, A. P. et al. Innovation and forward‐thinking are needed to improve traditional synthesis methods: a response to Pescott and Stewart. J. Appl. Ecol. 59, 1191–1197 (2022).

    Article  Google Scholar 

  68. Haddaway, N. R. et al. EviAtlas: a tool for visualising evidence synthesis databases. Environ. Evid. 8, 22 (2019).

  69. Delory, B. M., Li, M., Topp, C. N. & Lobet, G. archiDART v3.0: a new data analysis pipeline allowing the topological analysis of plant root systems. F1000Research 7, 22 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  70. Perkel, J. M. The future of scientific figures. Nature 554, 133–134 (2018).

    Article  CAS  PubMed  Google Scholar 

  71. Weaver, S. & Gleeson, M. P. The importance of the domain of applicability in QSAR modeling. J. Mol. Graph. Model. 26, 1315–1326 (2008).

    Article  CAS  PubMed  Google Scholar 

  72. Sutton, C. et al. Identifying domains of applicability of machine learning models for materials science. Nat. Commun. 11, 4428 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  73. Meyer, H. & Pebesma, E. Predicting into unknown space? Estimating the area of applicability of spatial prediction models. Methods Ecol. Evol. 12, 1620–1633 (2021).

    Article  Google Scholar 

  74. Pearl, J. & Bareinboim, E. Transportability of causal and statistical relations: a formal approach. In 2011 IEEE 11th International Conference on Data Mining Workshops https://doi.org/10.1109/ICDMW.2011.169 (IEEE, 2011).

  75. Munthe-Kaas, H., Nøkleby, H. & Nguyen, L. Systematic mapping of checklists for assessing transferability. Syst. Rev. 8, 22 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  76. Dekkers, O. M., von Elm, E., Algra, A., Romijn, J. A. & Vandenbroucke, J. P. How to assess the external validity of therapeutic trials: a conceptual approach. Int. J. Epidemiol. 39, 89–94 (2010).

    Article  CAS  PubMed  Google Scholar 

  77. Schloemer, T. & Schröder-Bäck, P. Criteria for evaluating transferability of health interventions: a systematic review and thematic synthesis. Implement. Sci. 13, 88 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  78. Fernandez-Hermida, J. R., Calafat, A., Becoña, E., Tsertsvadze, A. & Foxcroft, D. R. Assessment of generalizability, applicability and predictability (GAP) for evaluating external validity in studies of universal family-based prevention of alcohol misuse in young people: systematic methodological review of randomized controlled trials. Addiction 107, 1570–1579 (2012).

    Article  PubMed  Google Scholar 

  79. Avellar, S. A. et al. External validity: the next step for systematic reviews? Eval. Rev. 41, 283–325 (2017).

    Article  PubMed  Google Scholar 

  80. Bareinboim, E. & Pearl, J. A general algorithm for deciding transportability of experimental results. J. Causal Inference 1, 107–134 (2013).

    Article  Google Scholar 

  81. Degtiar, I. & Rose, S. A review of generalizability and transportability. Preprint at https://doi.org/10.48550/arXiv.2102.11904 (2021).

  82. Bareinboim, E. & Pearl, J. Meta-transportability of causal effects: a formal approach. J. Mach. Learn. Res. 31, 135–143 (2013).

    Google Scholar 

  83. Jamieson, D. Scientific uncertainty: how do we know when to communicate research findings to the public? Sci. Total Environ. 184, 103–107 (1996).

    Article  CAS  Google Scholar 

  84. Burchett, H. E. D., Mayhew, S. H., Lavis, J. N. & Dobrow, M. J. When can research from one setting be useful in another? Understanding perceptions of the applicability and transferability of research. Health Promot. Int. 28, 418–430 (2013).

    Article  PubMed  Google Scholar 

  85. Forscher, P. et al. Build up big-team science. Nature 601, 505–507 (2022).

    Article  Google Scholar 

  86. Whalen, M. A. et al. Climate drives the geography of marine consumption by changing predator communities. Proc. Natl Acad. Sci. USA 117, 28160–28166 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  87. Moshontz, H. et al. The Psychological Science Accelerator: advancing psychology through a distributed collaborative network. Adv. Methods Pract. Psychol. Sci. 1, 501–515 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  88. Marschner, I. C. A general framework for the analysis of adaptive experiments. Stat. Sci. 36, 465–492 (2021).

    Article  Google Scholar 

  89. Clark, M. Shrinkage in Mixed Effects Models https://m-clark.github.io/posts/2019-05-14-shrinkage-in-mixed-models/ (2019).

  90. Gurevitch, J. & Hedges, L. V. Statistical issues in ecological meta-analyses. Ecology 80, 1142–1149 (1999).

    Article  Google Scholar 

  91. Mengersen, K., Gurevitch, J. & Schmid, C. H. in Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, U. et al.) 300–312 (Princeton Univ. Press, 2013).

  92. Hudson, L. N. et al. The database of the PREDICTS (Projecting Responses of Ecological Diversity In Changing Terrestrial Systems) project. Ecol. Evol. 7, 145–188 (2017).

    Article  PubMed  Google Scholar 

  93. Dornelas, M. et al. BioTIME: a database of biodiversity time series for the Anthropocene. Glob. Ecol. Biogeogr. 27, 760–786 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  94. Salguero-Gómez, R. et al. The COMPADRE Plant Matrix Database: an open online repository for plant demography. J. Ecol. 103, 202–218 (2015).

    Article  Google Scholar 

  95. Salguero-Gómez, R. et al. COMADRE: a global data base of animal demography. J. Anim. Ecol. 85, 371–384 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  96. Pastor, D. A. & Lazowski, R. A. On the multilevel nature of meta-analysis: a tutorial, comparison of software programs, and discussion of analytic choices. Multivar. Behav. Res. 53, 74–89 (2018).

    Article  Google Scholar 

Download references

Acknowledgements

We thank J. Chase for informative discussion of concepts. R.S. is grateful for funding from the German Centre for Integrative Biodiversity Research – iDiv - Halle-Jena-Leipzig. J.M.B. was funded under UKCEH National Capability project 06895. C.T.C. was supported by a Marie Skłodowska-Curie Individual Fellowship (no. 891052).

Author information

Authors and Affiliations

Authors

Contributions

R.S. conceived the idea and developed a first draft with J.M.B., R.E.O., S.N., C.P.D., M.R. and C.T.C. contributed to idea development and paper writing.

Corresponding author

Correspondence to Rebecca Spake.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Ecology & Evolution thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Appendix S1 is a table of the aims and scope of the 50 most cited journals in ecology. Appendix S2 provides details of an analysis (a meta-regression of sapling abundance on thinning intensity in coniferous forests of Japan).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Spake, R., O’Dea, R.E., Nakagawa, S. et al. Improving quantitative synthesis to achieve generality in ecology. Nat Ecol Evol 6, 1818–1828 (2022). https://doi.org/10.1038/s41559-022-01891-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41559-022-01891-z

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing