Wang Bowen, Huggins-Manley Corinne, Kuang Huan, Xiong Jiawei
University of Florida, Gainesville, USA.
Florida State University, Tallahassee, USA.
Educ Psychol Meas. 2024 Oct 6:00131644241280727. doi: 10.1177/00131644241280727.
Rapid-guessing behavior in data can compromise our ability to estimate item and person parameters accurately. Consequently, it is crucial to model data with rapid-guessing patterns in a way that can produce unbiased ability estimates. This study proposes and evaluates three alternative modeling approaches that follow the logic of the effort-moderated item response theory model (EM-IRT) to analyze response data with rapid-guessing responses. One is the two-step EM-IRT model, which utilizes the item parameters estimated by respondents without rapid-guessing behavior and was initially proposed by Rios and Soland without further investigation. The other two models are effort-moderated multidimensional models (EM-MIRT), which we introduce in this study and vary as both between-item and within-item structures. The advantage of the EM-MIRT model is to account for the underlying relationship between rapid-guessing propensity and ability. The three models were compared with the traditional EM-IRT model regarding the accuracy of parameter recovery in various simulated conditions. Results demonstrated that the two-step EM-IRT and between-item EM-MIRT model consistently outperformed the traditional EM-IRT model under various conditions, with the two-step EM-IRT estimation generally delivering the best performance, especially for ability and item difficulty parameters estimation. In addition, different rapid-guessing patterns (i.e., difficulty-based, changing state, and decreasing effort) did not affect the performance of the two-step EM-IRT model. Overall, the findings suggest that the EM-IRT model with the two-step parameter estimation method can be applied in practice for estimating ability in the presence of rapid-guessing responses due to its accuracy and efficiency. The between-item EM-MIRT model can be used as an alternative model when there is no significant mean difference in the ability estimates between examinees who exhibit rapid-guessing behavior and those who do not.
数据中的快速猜测行为可能会损害我们准确估计项目和人员参数的能力。因此，以能够产生无偏能力估计的方式对具有快速猜测模式的数据进行建模至关重要。本研究提出并评估了三种替代建模方法，这些方法遵循努力调节项目反应理论模型（EM-IRT）的逻辑，以分析具有快速猜测反应的反应数据。一种是两步EM-IRT模型，它利用没有快速猜测行为的受访者估计的项目参数，最初由里奥斯和索兰德提出，但没有进一步研究。另外两种模型是努力调节多维模型（EM-MIRT），我们在本研究中引入了该模型，并且在项目间和项目内结构上有所不同。EM-MIRT模型的优点是考虑了快速猜测倾向与能力之间的潜在关系。在各种模拟条件下，将这三种模型与传统的EM-IRT模型在参数恢复准确性方面进行了比较。结果表明，两步EM-IRT模型和项目间EM-MIRT模型在各种条件下始终优于传统的EM-IRT模型，两步EM-IRT估计通常表现最佳，尤其是在能力和项目难度参数估计方面。此外，不同的快速猜测模式（即基于难度、状态变化和努力程度降低）不会影响两步EM-IRT模型的性能。总体而言，研究结果表明，具有两步参数估计方法的EM-IRT模型因其准确性和效率，可在实际中用于估计存在快速猜测反应时的能力。当表现出快速猜测行为的考生和没有表现出快速猜测行为的考生在能力估计上没有显著平均差异时，项目间EM-MIRT模型可以用作替代模型。
