Yuan Jianlin, Xiao Yue, Liu Hongyun
Educational Science Research Institute, Hunan University, Changsha, Hunan, China.
Faculty of Psychology, Beijing Normal University, Beijing, China.
Front Psychol. 2019 Feb 26;10:369. doi: 10.3389/fpsyg.2019.00369. eCollection 2019.
As one of the important 21st-century skills, collaborative problem solving (CPS) has aroused widespread concern in assessment. To measure this skill, two initiative approaches have been created: the human-to-human and human-to-agent modes. Between them, the human-to-human interaction is much closer to the real-world situation and its process stream data can reveal more details about the cognitive processes. The challenge for fully tapping into the information obtained from this mode is how to extract and model indicators from the data. However, the existing approaches have their limitations. In the present study, we proposed a new paradigm for extracting indicators and modeling the dyad data in the human-to-human mode. Specifically, both individual and group indicators were extracted from the data stream as evidence for demonstrating CPS skills. Afterward, a within-item multidimensional Rasch model was used to fit the dyad data. To validate the paradigm, we developed five online tasks following the asymmetric mechanism, one for practice and four for formal testing. Four hundred thirty-four Chinese students participated in the assessment and the online platform recorded their crucial actions with time stamps. The generated process stream data was handled with the proposed paradigm. Results showed that the model fitted well. The indicator parameter estimates and fitting indexes were acceptable, and students were well differentiated. In general, the new paradigm of extracting indicators and modeling the dyad data is feasible and valid in the human-to-human assessment of CPS. Finally, the limitations of the current study and further research directions are discussed.
作为21世纪的重要技能之一，协作问题解决（CPS）在评估中引起了广泛关注。为了测量这项技能，已经创建了两种主动方法：人对人模式和人对智能体模式。在这两种模式中，人对人互动更接近现实世界的情况，其过程流数据可以揭示更多关于认知过程的细节。充分利用从这种模式获得的信息面临的挑战是如何从数据中提取指标并进行建模。然而，现有方法存在局限性。在本研究中，我们提出了一种新的范式，用于在人对人模式下提取指标并对二元组数据进行建模。具体来说，从数据流中提取个体和小组指标，作为展示CPS技能的证据。随后，使用项目内多维Rasch模型来拟合二元组数据。为了验证该范式，我们按照非对称机制开发了五个在线任务，一个用于练习，四个用于正式测试。434名中国学生参与了评估，在线平台记录了他们带有时间戳的关键行为。生成的过程流数据采用所提出的范式进行处理。结果表明模型拟合良好。指标参数估计和拟合指数是可接受的，并且学生之间有很好的区分度。总体而言，在人对人CPS评估中，提取指标并对二元组数据进行建模的新范式是可行且有效的。最后，讨论了当前研究的局限性和进一步的研究方向。
