Olsen Erik, Fay Gavin, Gaichas Sarah, Gamble Robert, Lucey Sean, Link Jason S
Institute of Marine Research, PB 1870 Nordnes, N-5817, Bergen, Norway.
NOAA Northeast Fisheries Science Center, 166 Water St., Woods Hole, Massachusetts, 02543-1026, United States of America.
PLoS One. 2016 Jan 5;11(1):e0146467. doi: 10.1371/journal.pone.0146467. eCollection 2016.
Accelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted.
We assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes.
We identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).
全球生态系统的加速变化要求对过去、现在和未来在各种压力下的状态进行全面综合分析，以便充分了解当前和预计的未来系统状态。生态系统模型可为复杂多变环境中的人类活动管理提供信息，但这些模型可靠吗？确保模型对于解决管理问题可靠需要评估它们在表示现实世界过程和动态方面的技能。仅对有限的一些生物物理模型评估了技能。已经审查了一系列技能评估方法，但尚未尝试对完整的海洋生态系统模型进行技能评估。
我们通过将10年的模型预测与观测数据进行比较，评估了美国东北部（NEUS）亚特兰蒂斯海洋生态系统模型的技能。将模型预测性能与从40年的后报中获得的性能进行比较。使用多个指标（平均绝对误差、均方根误差、建模效率和斯皮尔曼等级相关性）以及一组时间序列（物种生物量、渔业上岸量和生态系统指标）来充分衡量模型技能。总体而言，NEUS模型表现高于平均水平，因此对于作为模型调整重点的关键物种而言，表现优于预期。模型预测技能与后报技能相当，表明模型性能在10年预测模式下不会退化，这是一个端到端生态系统模型对战略管理有用的重要特征。
我们确定了端到端生态系统模型技能评估的最佳实践方法，这将改善其他生态系统模型的实际应用以及未来的模型开发。我们表明，不仅有可能评估复杂海洋生态系统模型的技能，而且有必要这样做，以增强对模型结果的确信并鼓励将其用于战略管理。我们的方法适用于任何类型的预测模型，并且应考虑在生态学以外的领域（例如经济学、气候变化和风险评估）中使用。
