Department of Surgery, University of Manitoba, Canada.
Department of Surgery, University of Manitoba, Canada.
Am J Surg. 2020 Sep;220(3):610-615. doi: 10.1016/j.amjsurg.2020.01.019. Epub 2020 Jan 15.
Competency-based frameworks are common in surgical training. However, the optimal use of standardized technical assessments is not well defined. We investigated the effect of rater training (RT) on the reliability and validity of four assessment tools.
Forty-Seven surgeons were randomized to RT (N = 24) and no training (N = 23) groups. A task-specific checklist, pass-fail, visual analog, and OSATS global rating scale (GRS) were used to assess trainee knot-tying and suturing tasks. Delayed assessment was performed two weeks later. Internal consistency, intra/inter-rater reliability, and construct validity were measured.
The GRS had superior reliability and validity compared to the other tools regardless of training. No significant differences between training groups was found. However, the RT group trended to improved reliability for all tools at both assessments.
RT did not lead to significant improvements in skills assessments. Standardized assessments (OSATS GRS) are preferred due to their superior reliability and validity over other methods. Despite findings, we believe more effective training methods or repeated sessions may be required for sustained and significant effects.
基于能力的框架在外科培训中很常见。然而，标准化技术评估的最佳使用方法尚未明确。我们研究了评估者培训（RT）对四种评估工具的可靠性和有效性的影响。
将 47 名外科医生随机分为 RT（N=24）和无培训（N=23）组。使用特定于任务的检查表、通过/失败、视觉模拟和 OSATS 整体评估量表（GRS）来评估学员的打结和缝合任务。两周后进行延迟评估。测量内部一致性、内部/外部评估者可靠性和结构有效性。
无论是否接受培训，GRS 的可靠性和有效性均优于其他工具。培训组之间没有发现显著差异。然而，RT 组在两次评估中所有工具的可靠性均呈上升趋势。
RT 并未导致技能评估的显著改善。由于标准化评估（OSATS GRS）具有更高的可靠性和有效性，因此优于其他方法。尽管有这些发现，但我们认为可能需要更有效的培训方法或重复培训课程，以实现持续和显著的效果。
