School of Social and Political Sciences, University of Melbourne, Melbourne, Victoria, Australia.
Policy Institute, King's College London, London, Greater London, United Kingdom.
PLoS One. 2023 Aug 3;18(8):e0288469. doi: 10.1371/journal.pone.0288469. eCollection 2023.
The objective of this study is to investigate the application of machine learning techniques to the large-scale human expert evaluation of the impact of academic research. Using publicly available impact case study data from the UK's Research Excellence Framework (2014), we trained five machine learning models on a range of qualitative and quantitative features, including institution, discipline, narrative style (explicit and implicit), and bibliometric and policy indicators. Our work makes two key contributions. Based on the accuracy metric in predicting high- and low-scoring impact case studies, it shows that machine learning models are able to process information to make decisions that resemble those of expert evaluators. It also provides insights into the characteristics of impact case studies that would be favoured if a machine learning approach was applied for their automated assessment. The results of the experiments showed strong influence of institutional context, selected metrics of narrative style, as well as the uptake of research by policy and academic audiences. Overall, the study demonstrates promise for a shift from descriptive to predictive analysis, but suggests caution around the use of machine learning for the assessment of impact case studies.
本研究旨在探讨机器学习技术在大规模人类专家评估学术研究影响方面的应用。我们使用了英国卓越研究框架（2014 年）中公开提供的影响案例研究数据，针对包括机构、学科、叙述风格（明确和隐含）、计量学和政策指标在内的一系列定性和定量特征，对五个机器学习模型进行了训练。我们的工作有两个主要贡献。基于在预测高得分和低得分影响案例研究方面的准确性指标，它表明机器学习模型能够处理信息并做出类似于专家评估者的决策。它还提供了有关影响案例研究特征的见解，如果应用机器学习方法对其进行自动评估，这些特征将受到青睐。实验结果表明，机构背景、叙述风格的选定指标以及政策和学术受众对研究的接受度都具有很强的影响力。总的来说，该研究表明从描述性分析向预测性分析的转变具有很大的潜力，但也表明在使用机器学习评估影响案例研究时需要谨慎。
