Braithwaite Jeffrey, Herkes Jessica, Churruca Kate, Long Janet C, Pomare Chiara, Boyling Claire, Bierbaum Mia, Clay-Williams Robyn, Rapport Frances, Shih Patti, Hogden Anne, Ellis Louise A, Ludlow Kristiana, Austin Elizabeth, Seah Rebecca, McPherson Elise, Hibbert Peter D, Westbrook Johanna
Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Division of Health Sciences, University of South Australia, Adelaide, South Australia, Australia.
BMJ Open. 2019 Mar 30;9(3):e025320. doi: 10.1136/bmjopen-2018-025320.
Effective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.
We systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.
All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017. ELIGIBILITY CRITERIA: (1) English language, (2) published in the last 10 years (2007-2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher's achievements.
Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen's Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848-0.9039). Following screening, selected articles underwent full-text review and bias was assessed.
Four hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or 'altmetric' data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.
All metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.
有效的研究人员评估是资金分配、晋升和终身教职决策的关键。我们旨在确定关于评估研究人员成就方法的已知情况，从而形成一种新的综合评估模型。
我们通过系统评价与荟萃分析优先报告项目（PRISMA）框架对文献进行系统回顾。
截至2017年底的所有科学引文索引数据库（包括核心合集、MEDLINE和BIOSIS引文索引）。纳入标准：（1）英文文献；（2）过去10年（2007 - 2017年）发表；（3）有全文；（4）文章讨论了评估个体研究人员成就的方法。
文章被分配给四组评审人员进行筛选，每组随机抽取5%的文章同时对照纳入标准进行评审。使用科恩卡方系数（κ）评估评分者间信度。κ统计量显示一致性从中度到几乎完美（0.4848 - 至0.9039）。筛选后，对选定文章进行全文评审并评估偏倚。
最终评审纳入478篇文章。在我们纳入期之前开发的既定方法（如引文与产出、h指数和期刊影响因子）在文献和实践中仍然占主导地位。过去10年出现了新的文献计量方法和模型，包括：基于网页排名算法或“替代计量”数据的指标、应用同行评判的方法以及为出版物数量和质量赋值的技术。每种评估方法往往优先考虑成就的某些方面而非其他方面。
所有指标和模型都侧重于某些要素而牺牲了其他要素。本文提出了一种新的综合设计，即综合研究人员成就模型（CRAM），它取代了过去不合时宜的模型。CRAM可针对一系列应用进行修改。
