Belcher Brian M, Davel Rachel, Claus Rachel
Sustainability Research Effectiveness Program, College of Interdisciplinary Studies, Royal Roads University, Victoria V9B 5Y2, Canada.
Center for International Forestry Research, P.O. Box 0113 BOCBD, Bogor 16000, Indonesia.
MethodsX. 2020 Jan 23;7:100788. doi: 10.1016/j.mex.2020.100788. eCollection 2020.
With high and increasing expectations for research to have social and environmental impact, there is a corresponding need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes. Evaluation is especially challenging for problem-oriented research that employs inter- and transdisciplinary approaches and intervenes in complex systems, where experimental and statistical approaches to causal inference are inappropriate. Instead, theory-based evaluation can be applied to identify and test causal processes. This paper presents a detailed explanation of the Outcome Evaluation approach applied in Belcher et al. (2019b). It draws on concepts and approaches used in theory-based program evaluation and the more limited experience of theory-based research evaluation, providing a brief overview of conceptual strengths and limitations of other methods. The paper offers step-by-step guidance on application of the Outcome Evaluation approach, detailing how to: document a theory of change; determine data needs and sources; collect data; manage and analyze data; and present findings. This approach provides a clear conceptual and analytical framework in addition to actor-specific and impact pathway analyses for more precision in the assessment of outcomes. Specifically, the Outcome Evaluation approach: •Conceptualizes research within a complex system and explicitly recognizes the role of other actors, context, and external processes;•Utilizes a detailed actor-centred theory of change (ToC) as the analytical framework; and•Explicitly tests a set of hypotheses about the relationship between the research process/outputs and outcomes.
随着对研究产生社会和环境影响的期望不断提高且日益增长，相应地需要合适的方法来证明（用于问责）和分析（用于学习）研究项目是否以及如何促进变革过程。对于采用跨学科和多学科方法并干预复杂系统的问题导向型研究而言，评估尤其具有挑战性，因为实验和统计因果推断方法并不适用。相反，基于理论的评估可用于识别和检验因果过程。本文详细解释了贝尔彻等人（2019b）应用的结果评估方法。它借鉴了基于理论的项目评估中使用的概念和方法以及基于理论的研究评估中较为有限的经验，简要概述了其他方法在概念上的优势和局限性。本文提供了结果评估方法应用的分步指南，详细说明了如何：记录变革理论；确定数据需求和来源；收集数据；管理和分析数据；以及呈现研究结果。这种方法除了提供针对特定行为者和影响路径的分析以更精确地评估结果外，还提供了一个清晰的概念和分析框架。具体而言，结果评估方法：•在复杂系统中对研究进行概念化，并明确认识到其他行为者、背景和外部过程的作用；•将详细的以行为者为中心的变革理论（ToC）用作分析框架；•明确检验一组关于研究过程/产出与结果之间关系的假设。
