Department of Medical Informatics, Faculty of Medicine, RWTH Aachen University, Pauwelsstraße 30, 52074, Aachen, Germany.
Audiovisual Media Center, Faculty of Medicine, RWTH Aachen University, Aachen, Germany.
BMC Med Educ. 2022 Mar 16;22(1):186. doi: 10.1186/s12909-022-03231-y.
Reliable and objective assessment of psychomotor skills in physiotherapy students' education is essential for direct feedback and skill improvement. The aim of this study is to determine the interrater reliability in the assessment process of physiotherapy students and to analyse the assessment behaviour of the examiners.
Physiotherapy teachers from two different schools assessed students from two different schools performing proprioceptive neuromuscular facilitation (PNF) patterns. An evaluation sheet with a 6-point rating scale and 20 evaluation criteria including an overall rating was used for assessment. The interrater reliability was determined calculating an intraclass-correlation coefficient (ICC) and Krippendorff's alpha. The assessment behaviour of the examiners was further analysed calculating the location parameters and showing the item response distribution over item in form of a Likert plot.
The ICC estimates were mostly below 0.4, indicating poor interrater reliability. This was confirmed by Krippendorff's alpha. The examiners showed a certain central tendency and intergroup bias.
The interrater reliability in this assessment format was rather low. No difference between the two physiotherapy schools concerning the interrater reliability could be identified. Despite certain limitations of this study, there is a definite need for improvement of the assessment process in physiotherapy education to provide the students with reliable and objective feedback and ensure a certain level of professional competence in the students.
The study was approved by the ethics committee of the Medical Faculty RWTH Aachen University (EK 340/16).
在物理治疗学生的教育中，可靠和客观的心理运动技能评估对于直接反馈和技能提高至关重要。本研究的目的是确定物理治疗学生评估过程中的评分者间信度，并分析评分者的评估行为。
来自两所不同学校的物理治疗教师评估来自两所不同学校的学生进行本体感受神经肌肉促进（PNF）模式。评估使用带有 6 分制评分量表和 20 个评估标准（包括总体评分）的评估表进行。通过计算组内相关系数（ICC）和克里普多夫的 alpha 来确定评分者间信度。进一步通过计算位置参数并以李克特图的形式显示项目在项目中的反应分布来分析评分者的评估行为。
ICC 估计值大多低于 0.4，表明评分者间信度较差。这得到了克里普多夫的 alpha 的证实。评分者表现出一定的集中趋势和组间偏差。
在这种评估形式中，评分者间信度相当低。无法确定两所物理治疗学校在评分者间信度方面存在差异。尽管本研究存在一定的局限性，但确实需要改进物理治疗教育的评估过程，为学生提供可靠和客观的反馈，并确保学生具备一定的专业能力。
该研究得到了亚琛工业大学医学系伦理委员会的批准（EK 340/16）。
