Raftery James, Hanney Steve, Greenhalgh Trish, Glover Matthew, Blatch-Jones Amanda
Primary Care and Population Sciences, Faculty of Medicine, University of Southampton, Southampton General Hospital, Southampton, UK.
Health Economics Research Group (HERG), Institute of Environment, Health and Societies, Brunel University London, London, UK.
Health Technol Assess. 2016 Oct;20(76):1-254. doi: 10.3310/hta20760.
This report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.
(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.
We searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.
This narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.
The literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.
The findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence's remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities' research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish (researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.
There were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.
Research funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.
The National Institute for Health Research HTA programme.
本报告在2007年综述的基础上进行拓展，回顾了衡量研究项目影响的方法和工具。
（1）确定衡量健康研究项目影响的理论模型和实证方法的范围；（2）建立模型和方法的分类法；（3）总结这些模型应用和使用的证据；（4）评估卫生技术评估（HTA）项目的不同选项。
我们检索了包括Ovid MEDLINE、EMBASE、护理及相关健康文献累积索引和Cochrane图书馆在内的数据库，检索时间为2005年1月至2014年8月。
本叙述性系统文献综述包括更新、扩展以及分析/讨论。我们在2014年8月至2015年3月期间系统检索了八个数据库，并辅以个人知识。
关于影响评估的文献有了很大扩展。经调整后的回报框架仍然是使用最广泛的方法。它借鉴了不同的哲学传统，通过一个解释性案例研究元素和对背景的关注来强化潜在的逻辑模型。除了逻辑模型，其他理想类型的方法包括建构主义、实在论、批判和表演性方法。实际上，大多数模型都实用地借鉴了几种理想类型的元素。影响的货币化是一种越来越流行的方法，显示出研究的高回报，但严重依赖于关于健康收益在多大程度上依赖于研究的假设。尽管HTA项目通常在资助试验前需要进行系统综述，但它并没有常规检查这些试验对后续系统综述的影响。约克/以患者为中心的结果研究所和推荐分级评估、制定与评价工具包提供了评估这种影响的方法，但需要进行评估。如本综述所述，文献中很少有随机试验在阻止新技术使用方面发挥主要作用的实例。HTA项目资助的少数可能起到这种作用的试验是个例。
本综述的结果支持HTA项目继续使用回报框架。国民保健服务体系（NHS）结构的变化、英格兰NHS的发展以及国家卫生与临床优化研究所职责的变化，对识别和满足当前及未来的研究需求提出了新的挑战。未来对HTA项目影响的评估将不得不考虑更广泛的变化，特别是由于评估大学研究质量的卓越研究框架（REF）似乎可能继续依赖案例研究来衡量影响。HTA项目应考虑如何改进案例研究的形式和选择，以帮助进行更系统的评估。案例研究的选择，如在REF中，而且更普遍地，往往偏向于高影响而非低影响的事例。其他行业的经验表明，从后者可以学到很多东西。英国大多数主要研究资助者采用了Researchfish（英国剑桥的Researchfish有限公司），这对未来的影响评估有影响。虽然常规获取索引研究出版物有其优点，但Researchfish在收集其他非索引产出和活动方面的成功程度仍有待确定。
当我们将关注点从2007年综述的范围扩展到远远超出仅关注HTA项目的狭窄范围时，我们应对所面临挑战的程度存在局限性。
研究资助者可以从持续监测和评估他们资助的研究的影响中受益。他们还应审查案例研究的贡献，并扩大将试验与荟萃分析和指南联系起来的工作。
国家卫生研究院HTA项目。
