Peking University, Beijing, China.
BMC Med Educ. 2024 Aug 25;24(1):919. doi: 10.1186/s12909-024-05905-1.
Automated test assembly (ATA) represents a modern methodology that employs data science optimization on computer platforms to automatically create test form, thereby significantly improving the efficiency and accuracy of test assembly procedures. In the realm of medical education, large-scale high-stakes assessments often necessitate lengthy tests, leading to elevated costs in various dimensions (such as examinee fatigue and expenses associated with item development). This study aims to augment the design of the medical education assessments by leveraging modern ATA approaches.
To achieve the objective, a four-step process employing psychometric methodologies was used to calibrate and analyze the item pool of the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test comprising 300 multiple-choice questions (MCQ) in China. Subsequently, two modern ATA approaches were employed to determine the optimal item combination, accounting for both statistical and content requirements specified in the test blueprint. The qualities of the assembled test form, generated using modern ATA approaches, underwent meticulous evaluation.
Through an exploration of the psychometric properties of the SCTCMU as a foundational step, the evaluation revealed commendable quality in the item properties. Furthermore, the evaluation of the quality of assembled test form using modern ATA approaches indicated the ability to ascertain the optimal test length within the predefined measurement precision. Specifically, this investigation demonstrates that the application of modern ATA approaches can substantially reduce the test length of assembled test form, while simultaneously maintaining the required statistical and content standards specified in the test blueprint.
This study harnessed modern ATA approaches to facilitate the automatic construction of test form, thereby significantly enhancing the efficiency and precision of test assembly procedures. The utilization of modern ATA approaches offers medical educators a valuable tool to enhance the efficiency and cost-effectiveness of medical education assessment.
自动化测试组装（ATA）代表了一种现代方法，它利用计算机平台上的数据科学优化来自动创建测试形式，从而显著提高测试组装过程的效率和准确性。在医学教育领域，大规模高风险评估通常需要进行冗长的测试，从而导致在各个方面（如考生疲劳和与项目开发相关的费用）的成本增加。本研究旨在通过利用现代 ATA 方法来增强医学教育评估的设计。
为了实现目标，采用了四步过程，使用心理测量学方法来校准和分析中国全国性总结性测试——临床医学本科标准化能力测试（SCTCMU）的项目库，该测试由 300 个多项选择题（MCQ）组成。随后，采用两种现代 ATA 方法来确定最佳项目组合，同时考虑测试蓝图中规定的统计和内容要求。使用现代 ATA 方法组装的测试形式的质量经过了仔细的评估。
通过探索 SCTCMU 的心理测量特性作为基础步骤，评估结果显示项目特性质量良好。此外，使用现代 ATA 方法评估组装测试形式的质量表明，能够在预定义的测量精度内确定最佳测试长度。具体来说，这项研究表明，应用现代 ATA 方法可以大大缩短组装测试形式的测试长度，同时保持测试蓝图中规定的所需统计和内容标准。
本研究利用现代 ATA 方法来促进测试形式的自动构建，从而显著提高测试组装过程的效率和精度。现代 ATA 方法的使用为医学教育工作者提供了一个有价值的工具，以提高医学教育评估的效率和成本效益。
