Vaganay Arnaud
London School of Economics and Political Science, London, United Kingdom.
PLoS One. 2016 Sep 30;11(9):e0163702. doi: 10.1371/journal.pone.0163702. eCollection 2016.
The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light-especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or 'spin') in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government's commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform's champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended.
当某项改革未能兑现承诺时，评估结果的报告可能会成为评估者与政策制定者之间的争论焦点。评估者需要全面报告结果，而政策制定者则倾向于以积极的方式呈现这些结果——尤其是当他们此前曾表示支持该项改革时。目前的证据基础仅限于对政策评估者的一项调查、一项关于教育研究中报告偏倚的研究以及几项调查行业赞助对临床试验报告影响的研究。本研究有两个目标。其一，旨在使用临床医生制定的七个指标，评估试点评估报告中结果报告偏倚（ORB或“粉饰”）的风险。其二，试图考察政府对某项改革的支持承诺如何影响相应评估报告中的ORB水平。为回答这些问题，对13份评估报告进行了内容分析，所有这些报告均发现干预措施对其宣称的主要结果无显著影响。这些报告是从涵盖三个政策领域以及英国政府委托进行的13年研究的233项试点和实验评估数据集中系统选取的。结果表明，ORB风险是真实存在的。事实上，此处审查的所有研究都至少采用了一种与粉饰风险相关的呈现策略。本研究还发现，改革倡导者的资历与该改革评估中的ORB风险之间存在微弱的负相关。建议公布研究方案并使用报告指南。
