Institute of Medical Education Research Rotterdam, Erasmus MC, IMERR, Room AE-227, PO Box 2040, 3000CA, Rotterdam, the Netherlands.
Institute of Psychology, Erasmus University Rotterdam, PO Box 1738, 3000DR, Rotterdam, the Netherlands.
BMC Med Educ. 2018 Nov 26;18(1):282. doi: 10.1186/s12909-018-1390-0.
This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.
Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.
Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats.
Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.
本研究考察了两种情境判断测验（SJT）设计特征（反应指令和反应格式）对申请人认知的影响。此外，我们还研究了申请人对 SJT 的认知在人口统计学亚组中的差异。
医学院申请人（N=372）对申请人认知进行了在线调查，包括对 SJT 的描述和两个示例项目。受访者随机收到四种 SJT 版本之一（应做评分、应做选择一、将做评分、将做选择一）。他们对总体偏好以及四项程序正义因素（表面效度、申请人区分、与研究的相关性和表现机会）和作弊的难易程度进行了评分。此外，根据性别、族裔背景和第一代大学生身份对申请人认知进行了亚组比较。
申请人认为将做指令比应做指令更容易作弊。评分格式在申请人区分、与研究的相关性、表现机会和作弊的难易程度上比选择一格式获得了更有利的评价。在申请人认知方面，没有发现人口统计学亚组的显著主要效应，但显著的交互效应表明，某些亚组可能对某些 SJT 设计特征有更明显的偏好。具体来说，少数民族申请人——而不是多数族裔申请人——对应做指令的偏好大于将做指令。此外，第一代大学生——而不是非第一代大学生——对评分格式的偏好大于选择一格式。
研究结果表明，通过促进程序正义因素和降低感知作弊的容易程度，改变 SJT 设计特征可能会对申请人的认知产生积极影响，并且反应指令和反应格式可以提高 SJTs 对少数族裔申请人的吸引力。
