University Medical Center Hamburg-Eppendorf, Arbeitsgruppe Auswahlverfahren, Hamburg, Germany.
University of Heidelberg, Heidelberg, Germany.
GMS J Med Educ. 2024 Jun 17;41(3):Doc30. doi: 10.3205/zma001685. eCollection 2024.
Situational Judgement Tests (SJT) are a cost-efficient method for the assessment of personal characteristics (e.g., empathy, professionalism, ethical thinking) in medical school admission. Recently, complex open-ended response format SJTs have become more feasible to conduct. However, research on their applicability to a German context is missing. This pilot study tests the acceptability, reliability, subgroup differences, and validity of an online SJT with open-ended response format developed in Canada ("Casper").
German medical school applicants and students from Hamburg were invited to take Casper in 2020 and 2021. The test consisted of 12 video- and text-based scenarios, each followed by three open-ended questions. Participants subsequently evaluated their test experience in an online survey. Data on sociodemographic characteristics, other admission criteria (Abitur, TMS, HAM-Nat, HAM-SJT) and study success (OSCE) was available in a central research database (stav).
The full sample consisted of 582 participants. Test-takers' global perception of Casper was positive. Internal consistency was satisfactory in both years (=0.73; 0.82) while interrater agreement was moderate (ICC(1,2)=0.54). Participants who were female (=0.37) or did not have a migration background (=0.40) received higher scores. Casper scores correlated with HAM-SJT (=.18) but not with OSCE communication stations performance. The test was also related to Abitur grades (=-.15), the TMS (=.18), and HAM-Nat logical reasoning scores (=.23).
This study provides positive evidence for the acceptability, internal consistency, and convergent validity of Casper. The selection and training of raters as well as the scenario content require further observation and adjustments to a German context to improve interrater reliability and predictive validity.
情境判断测试（SJT）是一种评估医学院入学个人特征（如同理心、专业精神、伦理思维）的具有成本效益的方法。最近，复杂的开放式应答格式 SJT 变得更可行。然而，关于其在德国背景下适用性的研究尚缺。本初步研究测试了一种具有加拿大开发的开放式应答格式的在线 SJT（“Casper”）的可接受性、可靠性、亚组差异和有效性。
汉堡的德国医学院申请者和学生受邀于 2020 年和 2021 年参加 Casper 测试。测试由 12 个基于视频和文本的情景组成，每个情景后都有三个开放式问题。参与者随后在在线调查中评估他们的测试体验。社会人口统计学特征、其他入学标准（Abitur、TMS、HAM-Nat、HAM-SJT）和学业成功（OSCE）的数据可在中央研究数据库（stav）中获得。
总样本由 582 名参与者组成。应试者对 Casper 的总体看法是积极的。在这两年中，内部一致性都是令人满意的（=0.73；0.82），而评分者间的一致性是中度的（ICC（1,2）=0.54）。女性参与者（=0.37）或没有移民背景的参与者（=0.40）的得分较高。Casper 分数与 HAM-SJT 呈正相关（=.18），但与 OSCE 沟通站表现无关。该测试还与 Abitur 成绩（=-.15）、TMS（=0.18）和 HAM-Nat 逻辑推理分数（=0.23）相关。
本研究为 Casper 的可接受性、内部一致性和收敛效度提供了积极证据。评分者的选择和培训以及情景内容需要进一步观察和调整，以适应德国背景，提高评分者间的可靠性和预测效度。
