ALBERT

Description: Land models, which have been developed by the modeling community in the past two decades to predict future states of ecosystems and climate, have to be critically evaluated for their performance skills of simulating ecosystem responses and feedback to climate change. Benchmarking is an emerging procedure to measure and evaluate performance of models against a set of defined standards. This paper proposes a benchmarking framework for evaluation of land models. The framework includes (1) targeted aspects of model performance to be evaluated; (2) a set of benchmarks as defined references to test model performance; (3) metrics to measure and compare performance skills among models so as to identify model strengths and deficiencies; and (4) model improvement. Component 4 may or may not be involved in a benchmark analysis but is an ultimate goal of general modeling research. Land models are required to simulate exchange of water, energy, carbon and sometimes other trace gases between the atmosphere and the land-surface, and should be evaluated for their simulations of biophysical processes, biogeochemical cycles, and vegetation dynamics across timescales in response to both weather and climate change. Benchmarks that are used to evaluate models generally consist of direct observations, data-model products, and data-derived patterns and relationships. Metrics of measuring mismatches between models and benchmarks may include (1) a priori thresholds of acceptable model performance and (2) a scoring system to combine data-model mismatches for various processes at different temporal and spatial scales. The benchmark analyses should identify clues of weak model performance for future improvement. Iterations between model evaluation and improvement via benchmarking shall demonstrate progress of land modeling and help establish confidence in land models for their predictions of future states of ecosystems and climate.

Description: Terrestrial biosphere models (TBMs) have become an integral tool for extrapolating local observations and understanding of land–atmosphere carbon exchange to larger regions. The North American Carbon Program (NACP) Multi-scale synthesis and Terrestrial Model Intercomparison Project (MsTMIP) is a formal model intercomparison and evaluation effort focused on improving the diagnosis and attribution of carbon exchange at regional and global scales. MsTMIP builds upon current and past synthesis activities, and has a unique framework designed to isolate, interpret, and inform understanding of how model structural differences impact estimates of carbon uptake and release. Here we provide an overview of the MsTMIP effort and describe how the MsTMIP experimental design enables the assessment and quantification of TBM structural uncertainty. Model structure refers to the types of processes considered (e.g., nutrient cycling, disturbance, lateral transport of carbon), and how these processes are represented (e.g., photosynthetic formulation, temperature sensitivity, respiration) in the models. By prescribing a common experimental protocol with standard spin-up procedures and driver data sets, we isolate any biases and variability in TBM estimates of regional and global carbon budgets resulting from differences in the models themselves (i.e., model structure) and model-specific parameter values. An initial intercomparison of model structural differences is represented using hierarchical cluster diagrams (a.k.a. dendrograms), which highlight similarities and differences in how models account for carbon cycle, vegetation, energy, and nitrogen cycle dynamics. We show that, despite the standardized protocol used to derive initial conditions, models show a high degree of variation for GPP, total living biomass, and total soil carbon, underscoring the influence of differences in model structure and parameterization on model estimates.

Description: Significant changes in the water cycle are expected under current global environmental change. Robust assessment of these changes at global scales is confounded by shortcomings in the observed record. Modeled assessments yield conflicting results which are linked to differences in model structure and simulation protocol. Here we compare simulated runoff from six terrestrial biosphere models (TBMs), five reanalysis products, and one gridded surface station product with observations from a network of stream gauges in the contiguous United States (CONUS) from 2001 to 2005. We evaluate the consistency of simulated runoff with stream gauge data at the CONUS and water resource region scale, as well as examining similarity across TBMs and reanalysis products at the grid cell scale. Mean runoff across all simulated products and regions varies widely (range: 71–356 mm yr-1) relative to observed continental-scale runoff (209 mm yr-1). Across all 12 products only two are within 10% of the observed value and only four exhibit Nash–Sutcliffe efficiency values in excess of 0.8. Region-level mismatch exhibits a weak pattern of overestimation in western and underestimation in eastern regions; although two products are systematically biased across all regions. In contrast, bias in a temporal sense, within region by water year, is highly consistent. Although gridded composite TBM and reanalysis runoff show some regional similarities for 2001–2005 with CONUS means, individual product values are highly variable. To further constrain simulated runoff and to link model-observation mismatch to model structural characteristics would require watershed-level simulation studies coupled with river routing schemes, standardized forcing data, and explicit consideration of water cycle management.

Description: The El Niño Southern Oscillation is the dominant year-to-year mode of global climate variability. El Niño effects on terrestrial carbon cycling are mediated by associated climate anomalies, primarily drought, influencing fire emissions and biotic net ecosystem exchange (NEE). Here we evaluate whether El Niño produces a consistent response from the global carbon cycle. We apply a novel bottom-up approach to estimating global NEE anomalies based on FLUXNET data using land cover maps and weather reanalysis. We analyze 13 yr (1997–2009) of globally gridded observational NEE anomalies derived from eddy covariance flux data, remotely-sensed fire emissions at the monthly time step, and NEE estimated from an atmospheric transport inversion. We evaluate the overall consistency of biospheric response to El Niño and, more generally, the link between global CO2 flux anomalies and El Niño-induced drought. Our findings, which are robust relative to uncertainty in both methods and time-lags in response, indicate that each event has a different spatial signature with only limited spatial coherence in Amazônia, Australia and southern Africa. For most regions, the sign of response changed across El Niño events. Biotic NEE anomalies, across 5 El Niño events, ranged from −1.34 to +0.98 Pg C yr−1, whereas fire emissions anomalies were generally smaller in magnitude (ranging from −0.49 to +0.53 Pg C yr−1). Overall drought does not appear to impose consistent terrestrial CO2 flux anomalies during El Niños, finding large variation in globally integrated responses from −1.15 to +0.49 Pg C yr−1. Contrary to previous accounts we find El Niño events have, when globally integrated, both enhanced and weakened terrestrial sink strength, with no consistent response across events.

Description: Earth system processes exhibit complex patterns across time, as do the models that seek to replicate these processes. Model output may or may not be significantly related to observations at different times and on different frequencies. Conventional model diagnostics provide an aggregate view of model-data agreement, but usually do not identify the time and frequency patterns of model misfit, leaving unclear the steps required to improve model response to environmental drivers that vary on characteristic frequencies. Wavelet coherence can quantify the times and frequencies at which models and measurements are significantly different. We applied wavelet coherence to interpret the predictions of twenty ecosystem models from the North American Carbon Program (NACP) Site-Level Interim Synthesis when confronted with eddy covariance-measured net ecosystem exchange (NEE) from ten ecosystems with multiple years of available data. Models were grouped into classes with similar approaches for incorporating phenology, the calculation of NEE, and the inclusion of foliar nitrogen (N). Models with prescribed, rather than prognostic, phenology often fit NEE observations better on annual to interannual time scales in grassland, wetland and agricultural ecosystems. Models that calculate NEE as net primary productivity (NPP) minus heterotrophic respiration (HR) rather than gross ecosystem productivity (GPP) minus ecosystem respiration (ER) fit better on annual time scales in grassland and wetland ecosystems, but models that calculate NEE as GPP – ER were superior on monthly to seasonal time scales in two coniferous forests. Models that incorporated foliar nitrogen (N) data were successful at capturing NEE variability on interannual (multiple year) time scales at Howland Forest, Maine. Combined with previous findings, our results suggest that the mechanisms driving daily and annual NEE variability tend to be correctly simulated, but the magnitude of these fluxes is often erroneous, suggesting that model parameterization must be improved. Few NACP models correctly predicted fluxes on seasonal and interannual time scales where spectral energy in NEE observations tends to be low, but where phenological events, multi-year oscillations in climatological drivers, and ecosystem succession are known to be important for determining ecosystem function. Mechanistic improvements to models must be made to replicate observed NEE variability on seasonal and interannual time scales.

Description: Land models, which have been developed by the modeling community in the past few decades to predict future states of ecosystems and climate, have to be critically evaluated for their performance skills of simulating ecosystem responses and feedback to climate change. Benchmarking is an emerging procedure to measure performance of models against a set of defined standards. This paper proposes a benchmarking framework for evaluation of land model performances and, meanwhile, highlights major challenges at this infant stage of benchmark analysis. The framework includes (1) targeted aspects of model performance to be evaluated, (2) a set of benchmarks as defined references to test model performance, (3) metrics to measure and compare performance skills among models so as to identify model strengths and deficiencies, and (4) model improvement. Land models are required to simulate exchange of water, energy, carbon and sometimes other trace gases between the atmosphere and land surface, and should be evaluated for their simulations of biophysical processes, biogeochemical cycles, and vegetation dynamics in response to climate change across broad temporal and spatial scales. Thus, one major challenge is to select and define a limited number of benchmarks to effectively evaluate land model performance. The second challenge is to develop metrics of measuring mismatches between models and benchmarks. The metrics may include (1) a priori thresholds of acceptable model performance and (2) a scoring system to combine data–model mismatches for various processes at different temporal and spatial scales. The benchmark analyses should identify clues of weak model performance to guide future development, thus enabling improved predictions of future states of ecosystems and climate. The near-future research effort should be on development of a set of widely acceptable benchmarks that can be used to objectively, effectively, and reliably evaluate fundamental properties of land models to improve their prediction performance skills.

Description: The El Niño Southern Oscillation is the dominant year-to-year mode of global climate variability. El Niño effects on terrestrial carbon cycling are mediated by associated climate anomalies, primarily drought, influencing fire emissions and biotic net ecosystem exchange (NEE). Here we evaluate whether El Niño produces a consistent response from the global carbon cycle. We apply a novel bottom-up approach to estimating global NEE anomalies based on FLUXNET data using land cover maps and weather reanalysis. We analyze 13 years (1997–2009) of globally gridded observational NEE anomalies derived from eddy covariance flux data, remotely-sensed fire emissions at the monthly time step, and NEE estimated from an atmospheric transport inversion. We evaluate the overall consistency of biospheric response to El Niño and, more generally, the link between global CO2 flux anomalies and El Niño-induced drought. Our findings, which are robust relative to uncertainty in both methods and time-lags in response, indicate that each event has a different spatial signature with only limited spatial coherence in Amazônia, Australia and southern Africa. For most regions, the sign of response changed across El Niño events. Biotic NEE anomalies, across 5 El Niño events, ranged from –1.34 to +0.98 Pg C yr−1, whereas fire emissions anomalies were generally smaller in magnitude (ranging from –0.49 to +0.53 Pg C yr−1). Overall drought does not appear to impose consistent terrestrial CO2 flux anomalies during El Niños, finding large variation in globally integrated responses from –1.15 to +0.49 Pg C yr−1. Despite the significant correlation between the CO2 flux and El Niño indices, we find that El Niño events have, when globally integrated, both enhanced and weakened terrestrial sink strength, with no consistent response across events.

Description: Earth system processes exhibit complex patterns across time, as do the models that seek to replicate these processes. Model output may or may not be significantly related to observations at different times and on different frequencies. Conventional model diagnostics provide an aggregate view of model–data agreement, but usually do not identify the time and frequency patterns of model–data disagreement, leaving unclear the steps required to improve model response to environmental drivers that vary on characteristic frequencies. Wavelet coherence can quantify the times and timescales at which two time series, for example time series of models and measurements, are significantly different. We applied wavelet coherence to interpret the predictions of 20 ecosystem models from the North American Carbon Program (NACP) Site-Level Interim Synthesis when confronted with eddy-covariance-measured net ecosystem exchange (NEE) from 10 ecosystems with multiple years of available data. Models were grouped into classes with similar approaches for incorporating phenology, the calculation of NEE, the inclusion of foliar nitrogen (N), and the use of model–data fusion. Models with prescribed, rather than prognostic, phenology often fit NEE observations better on annual to interannual timescales in grassland, wetland and agricultural ecosystems. Models that calculated NEE as net primary productivity (NPP) minus heterotrophic respiration (HR) rather than gross ecosystem productivity (GPP) minus ecosystem respiration (ER) fit better on annual timescales in grassland and wetland ecosystems, but models that calculated NEE as GPP minus ER were superior on monthly to seasonal timescales in two coniferous forests. Models that incorporated foliar nitrogen (N) data were successful at capturing NEE variability on interannual (multiple year) timescales at Howland Forest, Maine. The model that employed a model–data fusion approach often, but not always, resulted in improved fit to data, suggesting that improving model parameterization is important but not the only step for improving model performance. Combined with previous findings, our results suggest that the mechanisms driving daily and annual NEE variability tend to be correctly simulated, but the magnitude of these fluxes is often erroneous, suggesting that model parameterization must be improved. Few NACP models correctly predicted fluxes on seasonal and interannual timescales where spectral energy in NEE observations tends to be low, but where phenological events, multi-year oscillations in climatological drivers, and ecosystem succession are known to be important for determining ecosystem function. Mechanistic improvements to models must be made to replicate observed NEE variability on seasonal and interannual timescales.

Description: Ecosystems are important and dynamic components of the global carbon cycle, and terrestrial biospheric models (TBMs) are crucial tools in further understanding of how terrestrial carbon is stored and exchanged with the atmosphere across a variety of spatial and temporal scales. Improving TBM model skills, and quantifying and reducing their estimation uncertainties, pose significant challenges. The Multi-scale Synthesis and Terrestrial Model Intercomparison Project (MsTMIP) is a formal multi-scale and multi-model intercomparison effort set up to tackle these challenges. The MsTMIP protocol prescribes standardized environmental driver data that are shared among model teams to facilitate model-model and model-observation comparisons. This paper describes the global and North American environmental driver data sets prepared for the MsTMIP activity to both support their use in MsTMIP and make these data, along with the processes used in selecting/processing these data, accessible to a broader audience. Based on project needs, we compiled climate, atmospheric CO2 concentrations, nitrogen deposition, land-use and land-cover change (LULCC), C3/C4 grasses fractions, major crops, phenology, and soil data into a standard format for global (0.5° x 0.5° resolution) and regional (North American, 0.25° x 0.25° resolution) simulations. In order to meet the needs of MsTMIP, improvements were made to several of the original environmental data sets, by changing the quality, the spatial and temporal coverage, resolution, or a combination of these. The resulting standardized model driver data sets are being used by over 20 different models participating MsTMIP. The data are archived at the Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC, http://daac.ornl.gov) to provide long-term data management and distribution.

Description: Terrestrial biosphere models (TBMs) have become an integral tool for extrapolating local observations and understanding of land-atmosphere carbon exchange to larger regions. The North American Carbon Program (NACP) Multi-scale synthesis and Terrestrial Model Intercomparison Project (MsTMIP) is a formal model intercomparison and evaluation effort focused on improving the diagnosis and attribution of carbon exchange at regional and global scales. MsTMIP builds upon current and past synthesis activities, and has a unique framework designed to isolate, interpret, and inform understanding of how model structural differences impact estimates of carbon uptake and release. Here we provide an overview of the MsTMIP effort and describe how the MsTMIP experimental design enables the assessment and quantification of TBM structural uncertainty. Model structure refers to the types of processes considered (e.g. nutrient cycling, disturbance, lateral transport of carbon), and how these processes are represented (e.g. photosynthetic formulation, temperature sensitivity, respiration) in the models. By prescribing a common experimental protocol with standard spin-up procedures and driver data sets, we isolate any biases and variability in TBM estimates of regional and global carbon budgets resulting from differences in the models themselves (i.e. model structure) and model-specific parameter values. An initial intercomparison of model structural differences is represented using hierarchical cluster diagrams (a.k.a. dendrograms), which highlight similarities and differences in how models account for carbon cycle, vegetation, energy, and nitrogen cycle dynamics. We show that, despite the standardized protocol used to derive initial conditions, models show a high degree of variation for GPP, total living biomass, and total soil carbon, underscoring the influence of differences in model structure and parameterization on model estimates.