From American Board of Internal Medicine, Philadelphia, Pennsylvania, and Uniformed Services University of the Health Sciences, Bethesda, Maryland.
Ann Intern Med. 2017 Sep 5;167(5):302-310. doi: 10.7326/M16-2843. Epub 2017 Aug 15.
Electronic resources are increasingly used in medical practice. Their use during high-stakes certification examinations has been advocated by many experts, but whether doing so would affect the capacity to differentiate between high and low abilities is unknown.
To determine the effect of electronic resources on examination performance characteristics.
Randomized controlled trial.
Medical certification program.
825 physicians initially certified by the American Board of Internal Medicine (ABIM) who passed the Internal Medicine Certification examination or sat for the Internal Medicine Maintenance of Certification (IM-MOC) examination in 2012 to 2015.
Participants were randomly assigned to 1 of 4 conditions: closed book using typical or additional time, or open book (that is, UpToDate [Wolters Kluwer]) using typical or additional time. All participants took the same modified version of the IM-MOC examination.
Primary outcomes included item difficulty (how easy or difficult the question was), item discrimination (how well the question differentiated between high and low abilities), and average question response time. Secondary outcomes included examination dimensionality (that is, the number of factors measured) and test-taking strategy. Item response theory was used to calculate question characteristics. Analysis of variance compared differences among conditions.
Closed-book conditions took significantly less time than open-book conditions (mean, 79.2 seconds [95% CI, 78.5 to 79.9 seconds] vs. 110.3 seconds [CI, 109.2 to 111.4 seconds] per question). Mean discrimination was statistically significantly higher for open-book conditions (0.34 [CI, 0.32 to 0.35] vs. 0.39 [CI, 0.37 to 0.41] per question). A strong single dimension showed that the examination measured the same factor with or without the resource.
Only 1 electronic resource was evaluated.
Inclusion of an electronic resource with time constraints did not adversely affect test performance and did not change the specific skill or factor targeted by the examination. Further study on the effect of resource inclusion on other examinations is warranted.
ABIM Foundation.
电子资源在医学实践中被越来越多地使用。许多专家主张在高风险认证考试中使用这些资源，但这样做是否会影响区分高能力和低能力的能力尚不清楚。
确定电子资源对考试表现特征的影响。
随机对照试验。
医学认证项目。
825 名最初由美国内科委员会（ABIM）认证的医生，他们在 2012 年至 2015 年期间通过了内科认证考试或参加了内科维持认证（IM-MOC）考试。
参与者被随机分配到以下 4 种条件之一：闭卷使用典型或额外时间，或开卷（即使用 UpToDate [Wolters Kluwer]）使用典型或额外时间。所有参与者都参加了相同的修改版 IM-MOC 考试。
主要结果包括项目难度（问题的难易程度）、项目区分度（问题区分高能力和低能力的程度）和平均问题响应时间。次要结果包括考试维度（即测量的因素数量）和考试策略。项目反应理论用于计算问题特征。方差分析比较了条件之间的差异。
闭卷条件的答题时间明显短于开卷条件（平均 79.2 秒[95%置信区间，78.5 秒至 79.9 秒]与 110.3 秒[置信区间，109.2 秒至 111.4 秒]）。开卷条件的平均区分度显著较高（0.34 [置信区间，0.32 至 0.35]与 0.39 [置信区间，0.37 至 0.41]）。一个强烈的单一维度表明，考试使用或不使用资源测量的是相同的因素。
仅评估了 1 种电子资源。
在有时间限制的情况下纳入电子资源不会对考试成绩产生不利影响，也不会改变考试所针对的特定技能或因素。有必要进一步研究资源纳入对其他考试的影响。
ABIM 基金会。
