Curran Megan L, Martin Emma E, Thomas Erin C, Singh Rashmi, Armana Saima, Kauser Asnia, Zaheer Eesha A, Sherry David D
University of Colorado School of Medicine, Children's Hospital of Colorado, 13123 E 16th Avenue, Box 311, Aurora, CO, 80045, USA.
Ann & Robert H. Lurie Children's Hospital of Chicago, 225 E. Chicago Avenue, Box 50, Chicago, IL, 60611, USA.
Pediatr Rheumatol Online J. 2019 Feb 8;17(1):5. doi: 10.1186/s12969-019-0308-7.
Of 37 pediatric rheumatology fellowship training programs in the United States, many have three or fewer fellows at a given time, making large-scale assessment of fellow performance difficult. An objective structured clinical examination (OSCE) is a scenario-based simulation method that assesses individual performance, thus indirectly measuring training program effectiveness. This study describes the development and implementation of two national pediatric rheumatology OSCEs and methods used for programmatic improvement.
OSCEs for pediatric rheumatology fellows were held in 2009 and 2011 during national rheumatology meetings using scenarios and assessment forms originally developed by a fellowship program director. The seven scenarios tested medical knowledge, physical exam and interpersonal skills. Pediatric rheumatologist evaluators assessed fellows' performance using checklists and gave immediate feedback. Program directors were sent summaries of their fellows' performances. Fellows evaluated the OSCE, providing organizational and scenario improvement suggestions. Programmatic changes to the 2011 OSCE were based on 2009 performance data and program evaluation feedback.
Twenty-two fellows participated in 2009 and 19 in 2011. Performance scores in similar scenarios did not change considerably over the two iterations. In 2009, 85.7% of participants reported desire to change clinical behavior. Assessors' 2009 program evaluation data prompted changes in rating scales and removal of invalid or unreliable assessments. Negative evaluation data about individual stations decreased from 60% in 2009 to 15.4% in 2011. Fellows' ratings of the experience's overall value were similar in 2009 and 2011. The average experience ratings were lower among fellows who proposed scenario-specific improvements and higher among those who recommended organizational improvements.
The 2011 examination exhibited programmatic improvement via reduction in fellows' scenario-specific negative feedback. Fellows' overall satisfaction did not change. Further work in scenario selection, assessment validation and inter-rater reliability will improve future pediatric rheumatology OSCEs.
在美国的37个儿科风湿病 fellowship 培训项目中，许多项目在特定时间的学员人数为三人或更少，这使得对学员表现进行大规模评估变得困难。客观结构化临床考试（OSCE）是一种基于情景的模拟方法，用于评估个人表现，从而间接衡量培训项目的有效性。本研究描述了两项全国性儿科风湿病 OSCE 的开发与实施以及用于项目改进的方法。
2009年和2011年在全国风湿病会议期间为儿科风湿病学员举办了 OSCE，使用的情景和评估表最初由一个 fellowship 项目主任制定。七个情景测试了医学知识、体格检查和人际技能。儿科风湿病专家评估人员使用检查表评估学员的表现并立即给予反馈。项目主任收到了其学员表现的总结。学员对 OSCE 进行了评估，提出了组织和情景改进建议。2011年 OSCE 的项目变更基于2009年的表现数据和项目评估反馈。
2009年有22名学员参加，2011年有19名学员参加。在两个轮次中，类似情景下的表现分数没有显著变化。2009年，85.7%的参与者表示希望改变临床行为。评估人员2009年的项目评估数据促使评分量表发生变化，并删除了无效或不可靠的评估。关于各个站点的负面评估数据从2009年的60%降至2011年的15.4%。2009年和2011年学员对该体验总体价值的评分相似。在提出情景特定改进建议的学员中，平均体验评分较低，而在建议组织改进的学员中，平均体验评分较高。
2011年的考试通过减少学员针对情景的负面反馈展示了项目改进。学员的总体满意度没有变化。在情景选择、评估验证和评估者间信度方面的进一步工作将改进未来的儿科风湿病 OSCE。
