Lertsakulbunlue Sethapong, Kantiwong Anupong
Department of Pharmacology, Phramongkutklao College of Medicine, Bangkok, 10400 Thailand.
Med Sci Educ. 2024 Sep 18;35(1):205-217. doi: 10.1007/s40670-024-02167-w. eCollection 2025 Feb.
The novel Immediate Self-feedback Very Short Answer Question (ISF-VSAQ) format exhibited testlet effect due to its instant feedback feature. This study utilized the testlet-as-a-polytomous-item-modeling (TPIM), an extension of item response theory, to tackle the testlet-related challenges.
Ninety-four pre-clinical students participated in two ten-item ISF-VSAQ exams on cardiovascular drugs based on a parallel blueprint. Each question involves clinical vignettes followed by expected answers, self-marking, and understanding levels. Messick's validity framework guided the collection of validity evidence.
Validity evidence included five sources: (1) Content: Experts reviews and alignment with the standard examination blueprint. (2) Response process: Students received an example and guide of ISF-VSAQ. Self-assessment steps were also explained, and all students were provided adequate time. (3) Internal structure: Higher marginal reliability was demonstrated in the TPIM model. While unidimensionality was violated in the second ISF-VSAQ, the TPIM maintained unidimensionality. Item discrimination was generally acceptable, except for one item on the second occasion with minor vignette changes. Items with new knowledge showed better discrimination and item information functions on the second occasion. (4) Relations to other variables: Path analysis shows that the first occasion's understanding and scoring abilities strongly influenced the second occasion, highlighting the testlet effect. (5) Consequences: Path analysis revealed that the understanding ability in the second attempt directly affected the summative cardiovascular ( = 0.26, = 0.066) and comprehensive examination ability ( = 0.34, = 0.010).
The TPIM model effectively tackles the testlet effect of the present ISF-VSAQ. Thus, when employing multiple instances of self-feedback format, analyzing it as a testlet is recommended.
The online version contains supplementary material available at 10.1007/s40670-024-02167-w.
新型即时自我反馈非常简短回答问题（ISF-VSAQ）格式因其即时反馈功能而呈现出组块效应。本研究采用组块作为多分类项目建模（TPIM），这是项目反应理论的一种扩展，以应对与组块相关的挑战。
94名临床前学生参加了基于平行蓝图的两次关于心血管药物的十项ISF-VSAQ考试。每个问题都涉及临床病例 vignettes，随后是预期答案、自我评分和理解水平。梅西克的效度框架指导了效度证据的收集。
效度证据包括五个来源：（1）内容：专家评审以及与标准考试蓝图的一致性。（2）反应过程：学生收到ISF-VSAQ的示例和指南。自我评估步骤也得到了解释，并且为所有学生提供了足够的时间。（3）内部结构：TPIM模型显示出更高的边际信度。虽然第二次ISF-VSAQ违反了单维性，但TPIM保持了单维性。项目区分度总体上是可接受的，除了第二次有一个项目的病例 vignettes 有轻微变化。具有新知识的项目在第二次表现出更好的区分度和项目信息功能。（4）与其他变量的关系：路径分析表明，第一次的理解和评分能力对第二次有强烈影响，突出了组块效应。（5）后果：路径分析显示，第二次尝试中的理解能力直接影响了心血管系统总结性考试能力（β = 0.26，p = 0.066）和综合考试能力（β = 0.34，p = 0.010）。
TPIM模型有效地解决了当前ISF-VSAQ的组块效应。因此，当采用多个自我反馈格式实例时，建议将其作为一个组块进行分析。
在线版本包含可在10.1007/s40670-024-02167-w获取的补充材料。
