Harding Susan-Marie E, Griffin Patrick E
Susan-Marie Harding, Assessment Research Centre, Melbourne Graduate School of Education, the University of Melbourne Parkville, VIC 3010, Australia,
J Appl Meas. 2016;17(1):35-53.
This paper describes an approach to the assessment of human to human collaborative problem solving using a set of online interactive tasks completed by student dyads. Within the dyad, roles were nominated as either A or B and students selected their own roles. The question as to whether role selection affected individual student performance measures is addressed. Process stream data was captured from 3402 students in six countries who explored the problem space by clicking, dragging the mouse, moving the cursor and collaborating with their partner through a chat box window. Process stream data were explored to identify behavioural indicators that represented elements of a conceptual framework. These indicative behaviours were coded into a series of dichotomous items. These items represented actions and chats performed by students. The frequency of occurrence was used as a proxy measure of item difficulty. Then given a measure of item difficulty, student ability could be estimated using the difficulty estimates of the range of items demonstrated by the student. The Rasch simple logistic model was used to review the indicators to identify those that were consistent with the assumptions of the model and were invariant across national samples, language, curriculum and age of the student. The data were analysed using a one and two dimension, one parameter model. Rasch separation reliability, fit to the model, distribution of students and items on the underpinning construct, estimates for each country and the effect of role differences are reported. This study provides evidence that collaborative problem solving can be assessed in an online environment involving human to human interaction using behavioural indicators shown to have a consistent relationship between the estimate of student ability, and the probability of demonstrating the behaviour.
本文描述了一种评估人与人之间协作式问题解决能力的方法，该方法使用了一组由学生二人组完成的在线互动任务。在二人组中，角色被指定为A或B，学生自行选择自己的角色。本文探讨了角色选择是否会影响个体学生的绩效指标这一问题。从六个国家的3402名学生中收集了过程流数据，这些学生通过点击、拖动鼠标、移动光标以及通过聊天框窗口与伙伴协作来探索问题空间。对过程流数据进行了探索，以识别代表概念框架要素的行为指标。这些指示性的行为被编码为一系列二分项目。这些项目代表了学生执行的操作和聊天内容。出现的频率被用作项目难度的替代度量。然后，给定项目难度的度量，就可以使用学生展示的一系列项目的难度估计来估计学生的能力。使用Rasch简单逻辑模型来审查这些指标，以识别那些与模型假设一致且在不同国家样本、语言、课程和学生年龄之间保持不变的指标。使用一维和二维单参数模型对数据进行了分析。报告了Rasch区分可靠性、与模型的拟合度、支撑结构上学生和项目的分布、每个国家的估计以及角色差异的影响。本研究提供了证据，表明在涉及人与人互动的在线环境中，可以使用行为指标来评估协作式问题解决能力，这些行为指标在学生能力估计与展示该行为的概率之间显示出一致的关系。
