Ludden-Schlatter Alicia, Bunt Stephanie, Hanrahan Kate DuChene
Department of Family and Community Medicine, University of Missouri, Columbia, MO.
Department of Family Medicine, Roy J. and Lucille A. Carver College of Medicine, University of Iowa, Iowa City, IA.
Fam Med. 2025 Apr;57(4):268-275. doi: 10.22454/FamMed.2025.615731. Epub 2025 Feb 5.
Residencies train residents in procedures and assess their competency, but existing assessment tools have demonstrated poor reliability and have not been validated.
This mixed-methods study validated a shave biopsy checklist with family medicine and dermatology faculty at two academic centers. In each phase of the study, teaching faculty scored a video-recorded simulated procedure using the checklist, and investigators assessed content validity, interrater reliability, and accuracy.
In focus groups of nine family medicine and dermatology faculty, 16 of 18 checklist items met or surpassed 80% interrater reliability. Overall checklist reliability was 74%. Focus group surveys initially revealed insufficient content validity. Lowest performing items were removed, and then the follow-up content validity index (0.76) surpassed the required threshold (0.62). Twenty-one of 70 family medicine faculty completed a final survey, which showed a content validity index of 0.63, surpassing the required threshold of 0.42. Twelve of 70 family medicine faculty viewed and scored a simulated video-recorded procedure. Overall interrater reliability was 91% (Cohen's d=1.36). Fourteen of 16 checklist items demonstrated greater than or equal to 90% interrater reliability. Accuracy analysis revealed 67.9% correct responses in focus groups and 84.9% in final testing (simple t test, P<.001, Cohen's d=1.4).
This rigorously validated checklist demonstrates appropriate content validity, interrater reliability, and accuracy. Findings support use of this shave biopsy checklist as an objective mastery standard for medical education and as a tool for formative assessment of procedural competency.
住院医师培训项目对住院医师进行操作培训并评估其能力，但现有的评估工具可靠性较差且未经验证。
这项混合方法研究在两个学术中心对家庭医学和皮肤科教员使用的刮除活检检查表进行了验证。在研究的每个阶段，教员使用该检查表对视频记录的模拟操作进行评分，研究人员评估内容效度、评分者间信度和准确性。
在由9名家庭医学和皮肤科教员组成的焦点小组中，18项检查表项目中有16项达到或超过了80%的评分者间信度。检查表总体信度为74%。焦点小组调查最初显示内容效度不足。删除表现最差的项目后，后续内容效度指数（0.76）超过了所需阈值（0.62）。70名家庭医学教员中有21名完成了最终调查，结果显示内容效度指数为0.63，超过了所需阈值0.42。70名家庭医学教员中有12名观看了模拟视频记录操作并进行评分。总体评分者间信度为91%（科恩d值=1.36）。16项检查表项目中有14项的评分者间信度大于或等于90%。准确性分析显示，焦点小组中的正确回答率为67.9%，最终测试中的正确回答率为84.9%（简单t检验，P<0.001，科恩d值=1.4）。
这份经过严格验证的检查表显示出适当的内容效度、评分者间信度和准确性。研究结果支持将这份刮除活检检查表用作医学教育的客观掌握标准以及形成性评估操作能力的工具。
