Buljan Ivan, Pina David G, Mijatović Antonija, Marušić Ana
Department of Psychology, Faculty of Humanities and Social Sciences in Split, University of Split, Split, Croatia.
European Research Executive Agency, European Commission, Brussels, Belgium.
F1000Res. 2024 Sep 5;12:1216. doi: 10.12688/f1000research.139743.1. eCollection 2023.
In the evaluation of research proposals, reviewers are often required to provide their opinions using various forms of quantitative and qualitative criteria. In 2020, the European Commission removed, for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) funding scheme, the numerical scores from the individual evaluations but retained them in the consensus report. This study aimed to assess whether there were any differences in reviewer comments' linguistic characteristics after the numerical scoring was removed, compared to comments from 2019 when numerical scoring was still present. This was an observational study and the data were collected for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) evaluation reports from the calls of 2019 and 2020, for both individual and consensus comments and numerical scores about the quality of the research proposal on three evaluation criteria: Excellence, Impact and Implementation. All comments were analyzed using the Linguistic Inquiry and Word Count (LIWC) program. For both years, the comments for proposal's strengths were written in a style that reflects objectivity, clout, and positive affect, while in weaknesses cold and objective style dominated, and that pattern remained stable across proposal status and research domains. Linguistic variables explained a very small proportion of the variance of the differences between 2019 and 2020 (McFadden R =0.03). Removing the numerical scores was not associated with the differences in linguistic characteristics of the reviewer comments. Future studies should adopt a qualitative approach to assess whether there are conceptual changes in the content of the comments.
在评估研究提案时，评审人员通常需要使用各种定量和定性标准来提供意见。2020年，欧盟委员会在玛丽·居里行动计划（MSCA）创新培训网络（ITN）资助计划中，取消了个人评估中的数值分数，但在共识报告中保留了这些分数。本研究旨在评估在取消数值评分后，评审意见的语言特征与2019年仍存在数值评分时的意见相比是否存在差异。这是一项观察性研究，收集了2019年和2020年玛丽·居里行动计划（MSCA）创新培训网络（ITN）评估报告的数据，包括关于研究提案质量的个人和共识意见以及在卓越性、影响力和实施这三个评估标准方面的数值分数。所有意见都使用语言查询与字数统计（LIWC）程序进行分析。在这两年中，关于提案优点的意见采用反映客观性、影响力和积极情感的风格撰写，而在缺点方面，冷淡和客观的风格占主导，并且这种模式在提案状态和研究领域中保持稳定。语言变量对2019年和2020年之间差异的方差解释比例非常小（麦克法登R = 0.03）。取消数值分数与评审意见的语言特征差异无关。未来的研究应采用定性方法来评估意见内容是否存在概念上的变化。
