Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia.
Computer Simulation & Advanced Research Technologies (CSART), Sydney, NSW, Australia.
PLoS One. 2022 Apr 22;17(4):e0266125. doi: 10.1371/journal.pone.0266125. eCollection 2022.
Systems modelling and simulation can improve understanding of complex systems to support decision making, better managing system challenges. Advances in technology have facilitated accessibility of modelling by diverse stakeholders, allowing them to engage with and contribute to the development of systems models (participatory modelling). However, despite its increasing applications across a range of disciplines, there is a growing need to improve evaluation efforts to effectively report on the quality, importance, and value of participatory modelling. This paper aims to identify and assess evaluation frameworks, criteria, and/or processes, as well as to synthesize the findings into a comprehensive multi-scale framework for participatory modelling programs.
A scoping review approach was utilized, which involved a systematic literature search via Scopus in consultation with experts to identify and appraise records that described an evaluation framework, criteria, and/or process in the context of participatory modelling. This scoping review is registered with the Open Science Framework.
The review identified 11 studies, which varied in evaluation purposes, terminologies, levels of examination, and time points. The review of studies highlighted areas of overlap and opportunities for further development, which prompted the development of a comprehensive multi-scale evaluation framework to assess participatory modelling programs across disciplines and systems modelling methods. The framework consists of four categories (Feasibility, Value, Change/Action, Sustainability) with 30 evaluation criteria, broken down across project-, individual-, group- and system-level impacts.
DISCUSSION & CONCLUSION: The presented novel framework brings together a significant knowledge base into a flexible, cross-sectoral evaluation effort that considers the whole participatory modelling process. Developed through the rigorous synthesis of multidisciplinary expertise from existing studies, the application of the framework can provide the opportunity to understand practical future implications such as which aspects are particularly important for policy decisions, community learning, and the ongoing improvement of participatory modelling methods.
系统建模和模拟可以提高对复杂系统的理解，以支持决策，更好地管理系统挑战。技术的进步使不同利益相关者更容易获得建模，使他们能够参与并为系统模型的开发做出贡献（参与式建模）。然而，尽管它在多个学科中的应用越来越广泛，但越来越需要改进评估工作，以有效地报告参与式建模的质量、重要性和价值。本文旨在确定和评估评估框架、标准和/或流程，并将研究结果综合为一个全面的参与式建模计划多尺度框架。
采用范围综述方法，通过 Scopus 进行系统文献检索，并与专家协商，以确定和评估描述参与式建模背景下评估框架、标准和/或流程的记录。本范围综述已在开放科学框架中注册。
该综述确定了 11 项研究，这些研究在评估目的、术语、检查级别和时间点上有所不同。对研究的回顾突出了重叠领域和进一步发展的机会，这促使开发了一个全面的多尺度评估框架，以评估跨学科和系统建模方法的参与式建模计划。该框架由四个类别（可行性、价值、变化/行动、可持续性）组成，有 30 个评估标准，分为项目、个人、小组和系统级影响。
提出的新框架将一个重要的知识库汇集在一起，形成一个灵活的、跨部门的评估工作，考虑了整个参与式建模过程。该框架是通过从现有研究中综合多学科专业知识而开发的，其应用有机会了解实际的未来影响，例如哪些方面对政策决策、社区学习和参与式建模方法的持续改进特别重要。
