Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center (LUMC), Leiden University, Leiden, the Netherlands.
Leiden University Neurosurgical Center Holland (UNCH), LUMC and The Hague Medical Center (HMC), Leiden, the Netherlands.
PLoS One. 2020 May 22;15(5):e0233318. doi: 10.1371/journal.pone.0233318. eCollection 2020.
Frameworks used in research impact evaluation studies vary widely and it remains unclear which methods are most appropriate for evaluating research impact in the field of surgical research. Therefore, we aimed to identify and review the methods used to assess the impact of surgical intervention trials on healthcare and to identify determinants for surgical impact.
We searched journal databases up to March 10, 2020 for papers assessing the impact of surgical effectiveness trials on healthcare. Two researchers independently screened the papers for eligibility and performed a Risk of Bias assessment. Characteristics of both impact papers and trial papers were summarized. Univariate analyses were performed to identify determinants for finding research impact, which was defined as a change in healthcare practice.
Sixty-one impact assessments were performed in 37 included impact papers. Some surgical trial papers were evaluated in more than one impact paper, which provides a total of 38 evaluated trial papers. Most impact papers were published after 2010 (n = 29). Medical records (n = 10), administrative databases (n = 22), and physician's opinion through surveys (n = 5) were used for data collection. Those data were analyzed purely descriptively (n = 3), comparing data before and after publication (n = 29), or through time series analyses (n = 5). Significant healthcare impact was observed 49 times and more often in more recent publications. Having impact was positively associated with using medical records or administrative databases (ref.: surveys), a longer timeframe for impact evaluation and more months between the publication of the trial paper and the impact paper, data collection in North America (ref.: Europe), no economic evaluation of the intervention, finding no significant difference in surgical outcomes, and suggesting de-implementation in the original trial paper.
Research impact evaluation receives growing interest, but still a small number of impact papers per year was identified. The analysis showed that characteristics of both surgical trial papers and impact papers were associated with finding research impact. We advise to collect data from either medical records or administrative databases, with an evaluation time frame of at least 4 years since trial publication.
研究影响评估研究中使用的框架差异很大，目前尚不清楚哪种方法最适合评估外科研究领域的研究影响。因此，我们旨在确定并审查评估外科干预试验对医疗保健影响的方法，并确定外科影响的决定因素。
我们检索了期刊数据库，截至 2020 年 3 月 10 日，以评估评估外科有效性试验对医疗保健影响的论文。两名研究人员独立筛选论文的合格性，并进行偏倚风险评估。总结了影响论文和试验论文的特征。进行单变量分析以确定发现研究影响的决定因素，研究影响定义为医疗保健实践的改变。
在 37 篇纳入的影响论文中进行了 61 项影响评估。一些外科试验论文在一篇以上的影响论文中进行了评估，这总共提供了 38 篇评估的试验论文。大多数影响论文发表于 2010 年之后（n = 29）。医疗记录（n = 10）、行政数据库（n = 22）和通过调查的医生意见（n = 5）用于数据收集。这些数据纯粹通过描述性分析（n = 3）、比较发表前后的数据（n = 29）或通过时间序列分析（n = 5）进行分析。观察到 49 次显著的医疗保健影响，并且在较新的出版物中更为常见。有影响与使用医疗记录或行政数据库（参考：调查）、更长的影响评估时间框架以及试验论文和影响论文发表之间的更多月份、北美数据收集（参考：欧洲）、干预的经济评估、手术结果无显著差异以及在原始试验论文中建议停止实施正相关。
研究影响评估越来越受到关注，但每年仍只确定了少量的影响论文。分析表明，外科试验论文和影响论文的特征都与发现研究影响有关。我们建议从医疗记录或行政数据库中收集数据，并在试验论文发表后至少 4 年进行评估。
