Moeller Jeremy, Gonzalez-Giraldo Ernesto, Fung France W, Johnson Emily L, Kheder Ammar, MacLean Jane, McGinnis Emily L, Muhlhofer Wolfgang G, Oster Joel M, Schmitt Sarah, Voinescu P Emanuela, Wong-Kisiel Lily C, Kidd Kandice J, Kephart Amy, Lado Fred A, Kessler Sudha Kilaru
From the Department of Neurology (J. Moeller), Yale School of Medicine, New Haven, CT; Departments of Neurology and Pediatrics (E.G.-G.), University of California, San Francisco; Departments of Neurology and Pediatrics (F.W.F., S.K.K.), Children's Hospital of Philadelphia and University of Pennsylvania Perelman School of Medicine, Philadelphia; Department of Neurology (E.L.J.), Johns Hopkins School of Medicine, Baltimore, MD; Department of Neurology and Pediatric Institute (A. Kheder), Emory University School of Medicine, Atlanta, GA; Department of Pediatric Neurology (J. MacLean), Sutter Medical Foundation, Mountain View, CA; Neurology Department (E.L.M.), Kaiser Permanente Los Angeles Medical Center, CA; University of Washington Regional Epilepsy Center (W.G.M.), University of Washington, Seattle; Department of Neurology (J.M.O.), Tufts University, Boston, MA; Department of Neurology (S.S.), Medical University of South Carolina, Charleston; Division of Epilepsy (P.E.V.), Department of Neurology, Brigham and Womens Hospital, Harvard Medical School, Boston, MA; Department of Neurology (L.C.W.-K.), Mayo Clinic College of Medicine, Rochester, MN; American Epilepsy Society (K.J.K., A. Kephart), Chicago, IL; and Department of Neurology (F.A.L.), Zucker School of Medicine at Hofstra-Northwell, Hempstead, NY.
Neurol Educ. 2023 Oct 26;2(4):e200090. doi: 10.1212/NE9.0000000000200090. eCollection 2023 Dec 22.
Epilepsy education has been transformed over the past 2 decades, leading to a need for structured formative assessment tools. The American Epilepsy Society developed the Epilepsy Fellowship In-Training Examination (EpiFITE) to provide high-quality formative assessment for fellows, to stimulate program improvement, and to guide future learning and teaching. The aim of this study was to explore validity evidence for the EpiFITE in meeting these goals.
Validity evidence was sought from multiple sources. The content of the examination was linked to the American Board of Psychiatry and Neurology blueprint for initial certification in epilepsy, and items were developed by trained experts. Internal structure was studied using internal consistency and item analysis. Surveys of fellows and fellowship directors focused on the examination experience (response process) and how results influenced fellow assessment, future learning, and program improvement (relationship to other variables and consequences).
The EpiFITE was first administered in 2020, with 172 examinees from 67 programs. By 2022 (year 3), the EpiFITE was completed by 195 epilepsy fellows from 77 programs. The overall mean score of the examination was stable from year to year, and the committee predicted the difficulty of individual items with a high degree of accuracy. The examination had high internal consistency (Cronbach α 0.76-0.81). The median item discrimination index ranged from 0.17 in 2020 to 0.21 in 2022. Discrimination indices were lower (mean ≤0.10) for items that were either very easy or very difficult and significantly higher (mean >0.20) for other items. Program directors and epilepsy fellows agreed the examination questions were appropriate and agreed that the EpiFITE helped them identify areas for self-directed learning. Program directors also found the examination helpful in identifying areas of strength and areas for improvement within their programs.
There are several sources of evidence of the quality and validity of the EpiFITE. By exploring this validity evidence, we have identified several best practices in the development and evaluation of a subspecialty examination, and this experience could be helpful for developers of in-training examinations in other subspecialties.
在过去20年里，癫痫教育已发生变革，这导致需要有结构化的形成性评估工具。美国癫痫协会开发了癫痫专科培训在职考试（EpiFITE），为专科培训学员提供高质量的形成性评估，以促进项目改进，并指导未来的学习与教学。本研究的目的是探索EpiFITE在实现这些目标方面的效度证据。
从多个来源寻求效度证据。考试内容与美国精神病学和神经病学委员会癫痫初始认证的蓝图相关联，题目由经过培训的专家编制。使用内部一致性和题目分析来研究内部结构。对学员和专科培训项目主任的调查聚焦于考试体验（作答过程）以及考试结果如何影响学员评估、未来学习和项目改进（与其他变量及结果的关系）。
EpiFITE于2020年首次施行，来自67个项目的172名考生参加了考试。到2022年（第3年），来自77个项目的195名癫痫专科培训学员完成了EpiFITE。考试的总体平均分逐年稳定，委员会对单个题目的难度预测具有高度准确性。该考试具有较高的内部一致性（克朗巴哈α系数为0.76 - 0.81）。题目区分度指数中位数从2020年的0.17到2022年的0.21不等。对于非常简单或非常难的题目，区分度指数较低（均值≤0.10），而对于其他题目则显著较高（均值>0.20）。项目主任和癫痫专科培训学员一致认为考试题目合适，并认为EpiFITE帮助他们确定了自主学习的领域。项目主任还发现该考试有助于确定其项目中的优势领域和改进领域。
有多个证据来源证明了EpiFITE的质量和效度。通过探索这一效度证据，我们确定了专科考试开发与评估中的几个最佳实践，这一经验可能对其他专科培训考试的开发者有所帮助。
