Testing ecological models: the meaning of validation

doi:10.1016/0304-3800(95)00152-2

Ecological Modelling

Volume 90, Issue 3, 1 November 1996, Pages 229-244

https://doi.org/10.1016/0304-3800(95)00152-2 Get rights and content

Abstract

The ecological literature reveals considerable confusion about the meaning of validation in the context of simulation models. The confusion arises as much from semantic and philosophical considerations as from the selection of validation procedures. Validation is not a procedure for testing scientific theory or for certifying the ‘truth’ of current scientific understanding, nor is it a required activity of every modelling project. Validation means that a model is acceptable for its intended use because it meets specified performance requirements.

Before validation is undertaken, (1) the purpose of the model, (2) the performance criteria, and (3) the model context must be specified. The validation process can be decomposed into several components: (1) operation, (2) theory, and (3) data. Important concepts needed to understand the model evaluation process are verification, calibration, validation, credibility, and qualification. These terms are defined in a limited technical sense applicable to the evaluation of simulation models, and not as general philosophical concepts. Different tests and standards are applied to the operational, theoretical, and data components. The operational and data components can be validated; the theoretical component cannot.

The most common problem with ecological and environmental models is failure to state what the validation criteria are. Criteria must be explicitly stated because there are no universal standards for selecting what test procedures or criteria to use for validation. A test based on comparison of simulated versus observed data is generally included whenever possible. Because the objective and subjective components of validation are not mutually exclusive, disagreements over the meaning of validation can only be resolved by establishing a convention.

References (51)

H. Caswell
The validation problem
H. Caswell
Theory and models in ecology: a different perspective
Ecol. Model.
(1988)
R. Costanza
Model goodness of fit: a multiple resolution procedure
Ecol. Model.
(1989)
S. Gentil et al.
Validation of complex ecosystems models
Ecol. Model.
(1981)
D.G. Mayer et al.
Statistical validation
Ecol. Model.
(1993)
R.V. O'Neill et al.
Multiple nutrient limitations in ecological models
Ecol. Model.
(1989)
M. Power
The predictive validation of ecological and environmental models
Ecol. Model.
(1993)
G.I. Ågren et al.
State-of-the-art models of production-decomposition linkages in conifer and grassland ecosystems
Ecol. Appl.
(1991)
J. Bart
Acceptance criteria for using individual-based models to make management decisions
Ecol. Appl.
(1995)
D.B. Botkin
Expert Panel Report 1

D.B. Botkin

Forest Dynamics

R.D. Cess et al.

Intercomparison and interpretation of climate feedback processes in 19 atmospheric general circulation models

J. Geophys. Res.

(1990)

G.L. Curry et al.

Discrete Simulation

C.R. Darwin

On the Origin of Species

(1859)

T. Fagerstrom

On theory, data and mathematics in ecology

Oikos

(1987)

G.S. Fishman et al.

The Statistics of Discrete-Event Simulation

Simulation

(1968)

R.A. Fleming et al.

Evaluating models for spruce budworm-forest management: comparing output with field data

Ecol. Appl.

(1992)

R.N. Giere

Understanding Scientific Reasoning

D.W. Goodall

Building and testing ecosystem models

C.S. Holling

Adaptive Environmental Assessment and Management

S.V. Hoover et al.

Simulation

J.N.R. Jeffers

An Introduction to Systems Analysis: With Ecological Applications

R. Jeffrey

Formal Logic

S.E. Jørgensen

Fundamentals of Ecological Modelling

J.P.C. Kleijnen

Statistical Tools for Simulation Practitioners

Cited by (920)

Testing temporal transferability of remote sensing models for large area monitoring
2024, Science of Remote Sensing
Applying remote sensing models outside the temporal range of their training data, referred to as temporal model transfer, has become common practice for large area monitoring projects that extrapolate models for hindcasting or forecasting to time periods lacking reference data. However, the development of appropriate validation methods for temporal transfer has lagged behind its rapid adoption. Breaking temporal transfer's assumption of temporal consistency in both remote sensing and reference data and their relationship to each other could lead to biased pixel-level predictions and small area estimators, compromising the operational validity of large area monitoring products. Few studies using temporal transfer have evaluated its effects on model accuracy at the pixel/plot level, and the propensity for biased small area estimators and trends resulting from temporal transfer remains unexplored. We present a framework for evaluating temporal transferability by combining temporal cross-validation with a multiscale map assessment to aid in identifying where and when biased model predictions could scale to small area estimates and affect predicted trends.
This validation framework is demonstrated in a case study of annual percent tree canopy cover mapping in Michigan. We tested and compared temporal transferability of random forest models of canopy cover derived from 2010 to 2016 systematic dot-grid photo-interpretations at Forest Inventory and Analysis plots with Landsat spectral indices fit with the LandTrendr temporal segmentation algorithm serving as the primary predictor variables. The temporal cross-validation error (mean RMSE = 13.9% cover) was higher than the common validation approach of considering all time periods of testing data together (RMSE = 13.6% cover) and lower than models trained and tested within the same year (mean RMSE = 14.2% cover). However, the bias of model predictions and small area estimators for individual years was higher with temporal transfer models than when applying models within the same year as their training data. We also evaluated how training models using different temporal subsets and with and without LandTrendr fitting affected predictions of Michigan's 1984–2020 predicted annual mean cover. The mean cover from LandTrendr-based models followed expected and consistent trends and had less difference between models trained with different temporal subsets (max difference = 5.8% cover). While those from Landsat had high interannual variations and greater difference between temporal models (max difference = 11.2% cover). The results of this case study demonstrate that evaluation of temporal transferability is necessary for establishing the operational validity of large area monitoring products, even when using time series algorithms that improve temporal consistency.
Conceptualization of a vehicle-to-grid assisted nation-wide renewable energy system – A case study with spain
2024, Energy Conversion and Management: X
This study explores the potential of Vehicle-to-Grid (V2G) technology in utilizing Electric Vehicle (EV) batteries for energy storage, aiming to fulfil Spain's 2030 and 2050 energy goals. The validated Simulink model uses 3.15 million EVs in 2030 and 22.7 million EVs in 2050 as primary energy storage. The results show that Spain can achieve its 2030 target of 42 % renewable energy by utilizing 800 GW of PV + Wind, or 220 GW of installations combined with an annual import of 18 TWh. For the 2050 goal of 97 % renewable energy, the study proposes 1300 GW installation without external support, or 600 GW renewable energy source installation with a 50 TWh import. Detailed analysis shows that the storage provided by 3.15 million EVs can replace 122 GW of new energy storage installation in 2030 and 22.7 million EVs replace 2.7 TW of new energy storage requirements in 2050 to support high penetration of intermittent renewable energy installations. The analysis showcases the substantial storage capacity provided by EVs, emphasizing their potential as a primary ESS and the effectiveness of V2G technology in grid support. The overall results underscore V2G technology's role in minimizing the need for additional energy storage infrastructure in the future.
Assessing impacts of a notorious invader (common carp Cyprinus carpio) on Australia's aquatic ecosystems: Coupling abundance-impact relationships with a spatial biomass model
2024, Biological Conservation
Quantifying the ecological impacts of non-native species is essential for large scale conservation management. Estimating such impacts often require spatially explicit abundance(/biomass) models of the target non-native species and knowledge of abundance-impact relationships, both of which are rarely known. Common carp (Cyprinus carpio) is a major threat to aquatic systems globally and relatively well-studied but yet models quantifying landscape-scale impacts of carp invasions are lacking. Here, we undertake a nonlinear meta-analytic approach to describe biomass-impact relationships for eight impact metrics and then integrate these relationships with a recently published spatial carp biomass model for Australia in order to quantify the ecological impacts of the carp invasion on Australian aquatic ecosystems. For the meta-analysis we collated 286 abundance(biomass)-impact estimates from 41 studies. Our meta-analysis identified both linear and nonlinear biomass-impact relationships across the impact metrics. Using a small, out-of-sample validation from carp removal experiments, the model had high prediction accuracy (within <20 % of estimate) for four metrics, but one metric (macrophyte recovery) was consistently underestimated by 10 to 35 %. Finally, we estimated that the carp invasion in Australia has resulted in decreases in macrophytes (median of −36 %) and macroinvertebrates (−31 %) and increases in nitrogen (2 %), plankton biomass (7 %), phosphorus (8 %) and turbidity (63 %). The invasive spread of carp has fundamentally altered Australia's aquatic environments and this study provides the first quantitative assessment of this impact and a tool to guide carp management decisions.
Postfire damage zoning with open low-density LiDAR data sources in semi-arid forests of the Iberian Peninsula
2024, Remote Sensing Applications: Society and Environment
Wildfires represent one of the major ecological disasters with significant repercussions for terrestrial ecosystems (loss of biodiversity, material damages, erosion, etc). Using LiDAR (Light Detection and Ranging) technology allows large amounts of forest measurements data to be obtained, such as size of volume, biomass, canopy cover, heights, among others. It also allows to quantify some damage caused by forest fires by comparing LiDAR data point clouds at two different times (before and after fire). By correlating in situ measurements with statistical parameters from point clouds, we aim to evaluate postfire deviations in volume, biomass and canopy integrity. The aim of this study was to estimate the losses caused by the 2012 wildfire in the Public Utility Forest No.82 “Sierra de los Donceles” (Hellín, Albacete, Spain), which devastated more than 5000 ha. We used 65 plots from the 4th National Forest Inventory (NFI4) and the LiDAR data from 2009 to develop statistical models and train Artificial Neural Networks (ANN). Another independent set of 29 plots (of 14.1-m radius with sub-metric GPS) was employed to validate the regression and ANN models. The regression results showed better performance for estimating heights (R² = 75.03%, RMSE = 28.90%), volume (R² = 70.32%, RMSE = 37.61%) and basal area (R² = 68.37%, RMSE = 37.18). The total biomass showed the best fit (R² = 93.51%), but also the maximum error (RMSE = 40.08%). The worst fit result was the estimation of number of trees per hectare (R2 = 33.76%, RMSE = 30.97%). All the models were improved with neural network implementation: number of trees per hectare (R² = 44.41%, RMSE = 26.74%), basal area (R² = 68.81%, RMSE = 33.49%), dominant height (R² = 75.55%, RMSE = 15.43%), volume (R² = 61.03%, RMSE = 14.78%) and biomass (R² = 99.21%, RMSE = 20.81%). With our top-performing models in the 2009- and 2016-point cloud data, we measured forest reductions in terms of trees per hectare, basal area, predominant height, volume and biomass. These models provided a comprehensive view of forest losses by highlighting major decreases in vegetation coverage, tree density and tree heights.
A systematic review of mechanistic models of riverine macrophyte growth
2024, Aquatic Botany
Riverine macrophytes play diverse and foundational ecological roles, directly influencing ecosystem properties from local biodiversity to flows of water, energy, nutrients, and sediment, many of which in turn are central to river management. Numerical modeling is thus a crucial tool for understanding macrophyte and ecosystem responses to environmental, ecological, or management changes. However, riverine macrophytes have received relatively limited modeling attention compared to plants in many other aquatic or terrestrial systems. We conducted a systematic review of riverine macrophyte growth models, focusing on mechanisms of macrophyte growth, biomass loss, and feedback effects on river ecosystems. Processes such as light availability, thermal tolerance, nutrient limitation, and mortality were widely included in almost all models meeting the review criteria. However, models varied widely in their inclusion of processes such as shading, scour, and the roles of macrophytes in stream nutrient cycles. There has been relatively little consideration of factors such as dispersal, carbon sources, herbivory, burial, desiccation, and competition for space or nutrients, indicating directions for future modeling work. In light of this, we present a conceptual framework to help guide future macrophyte growth modelers through a thorough consideration of macrophytes’ myriad interactions with their ecosystems. We also emphasize the importance of modularity and accessibility toward improving efforts to model, and in turn manage, riverine ecosystems.
A new modelling framework for predator-prey interactions: A case study of an aphid-ladybeetle system
2023, Ecological Informatics
The significance of the relationship between data and models in ecology cannot be overstated. Over the past few decades, numerous techniques have been developed and applied in the study of ecological systems. One such technique is the Bayesian calibration of model parameters, which draws from Bayes' theorem to incorporate prior knowledge in estimating parameter values and quantifying their uncertainty. In this study, we establish a Bayesian framework for analysing ecological time series and apply it to a specific case of aphid-ladybeetle predation. To accomplish this, we construct and assess six mathematical models consisting of ordinary differential equations. These models encompass three phenomenological models and three data-driven models. To gain insights into the behaviour of the models concerning a specific quantity of interest, namely aphid abundance, we conduct a sensitivity analysis of the parameters. Subsequently, we compare the outputs of the models by employing model selection techniques. Based on our analysis, the best-performing model is a phenomenological one that incorporates a Holling's type II functional response. We believe that this framework has the potential for application and extension to other ecological systems, thereby enhancing our understanding of these intricate systems.

View all citing articles on Scopus

^∗: Present address: Pacific Northwest Laboratory, P.O. Box 999, Richland, WA 99352, USA. Fax: (+ 1-509) 372-3515

View full text

Testing ecological models: the meaning of validation

Abstract

Ecol. Model.

Ecol. Model.

Ecol. Model.

Ecol. Model.

Ecol. Model.

Ecol. Model.

State-of-the-art models of production-decomposition linkages in conifer and grassland ecosystems

Ecol. Appl.

Acceptance criteria for using individual-based models to make management decisions

Ecol. Appl.

Expert Panel Report 1

Forest Dynamics

Intercomparison and interpretation of climate feedback processes in 19 atmospheric general circulation models

J. Geophys. Res.

Discrete Simulation

On the Origin of Species

On theory, data and mathematics in ecology

Oikos

The Statistics of Discrete-Event Simulation

Simulation

Evaluating models for spruce budworm-forest management: comparing output with field data

Ecol. Appl.

Understanding Scientific Reasoning

Building and testing ecosystem models

Adaptive Environmental Assessment and Management

Simulation

An Introduction to Systems Analysis: With Ecological Applications

Formal Logic

Fundamentals of Ecological Modelling

Statistical Tools for Simulation Practitioners