Institute of Medical Education, Health Science Center, Peking University, Haidian District, 38 Xueyuan Rd, Beijing, China.
National Center for Health Professions Education Development, Peking University, Beijing, China.
Behav Res Methods. 2024 Oct;56(7):8080-8090. doi: 10.3758/s13428-024-02472-7. Epub 2024 Jul 29.
Mixed-format tests, which typically include dichotomous items and polytomously scored tasks, are employed to assess a wider range of knowledge and skills. Recent behavioral and educational studies have highlighted their practical importance and methodological developments, particularly within the context of multivariate generalizability theory. However, the diverse response types and complex designs of these tests pose significant analytical challenges when modeling data simultaneously. Current methods often struggle to yield reliable results, either due to the inappropriate treatment of different types of response data separately or the imposition of identical covariates across various response types. Moreover, there are few software packages or programs that offer customized solutions for modeling mixed-format tests, addressing these limitations. This tutorial provides a detailed example of using a Bayesian approach to model data collected from a mixed-format test, comprising multiple-choice questions and free-response tasks. The modeling was conducted using the Stan software within the R programming system, with Stan codes tailored to the structure of the test design, following the principles of multivariate generalizability theory. By further examining the effects of prior distributions in this example, this study demonstrates how the adaptability of Bayesian models to diverse test formats, coupled with their potential for nuanced analysis, can significantly advance the field of psychometric modeling.
混合格式测试通常包括二分项目和多分评分任务，用于评估更广泛的知识和技能。最近的行为和教育研究强调了它们的实际重要性和方法学发展，特别是在多元概化理论的背景下。然而，这些测试的多样化响应类型和复杂设计在同时对数据进行建模时带来了重大的分析挑战。当前的方法往往难以产生可靠的结果，要么是因为对不同类型的响应数据分别进行了不当处理，要么是因为对各种响应类型施加了相同的协变量。此外，很少有软件包或程序提供针对混合格式测试建模的定制解决方案，以解决这些限制。本教程提供了一个详细的示例，介绍如何使用贝叶斯方法对包含多项选择和自由回答任务的混合格式测试收集的数据进行建模。该建模使用 R 编程系统中的 Stan 软件进行，Stan 代码根据测试设计的结构进行了定制，遵循多元概化理论的原则。通过进一步研究这个例子中的先验分布的影响，本研究展示了贝叶斯模型对不同测试格式的适应性，以及它们进行细致分析的潜力，如何能极大地推动心理测量建模领域的发展。
