Educational Measurement, IPN - Leibniz Institute for Science and Mathematics Education, Olshausenstraße 62, 24118, Kiel, Germany.
Educational Testing Service, Princeton, USA.
Psychometrika. 2021 Mar;86(1):190-214. doi: 10.1007/s11336-020-09743-0. Epub 2021 Feb 5.
Complex interactive test items are becoming more widely used in assessments. Being computer-administered, assessments using interactive items allow logging time-stamped action sequences. These sequences pose a rich source of information that may facilitate investigating how examinees approach an item and arrive at their given response. There is a rich body of research leveraging action sequence data for investigating examinees' behavior. However, the associated timing data have been considered mainly on the item-level, if at all. Considering timing data on the action-level in addition to action sequences, however, has vast potential to support a more fine-grained assessment of examinees' behavior. We provide an approach that jointly considers action sequences and action-level times for identifying common response processes. In doing so, we integrate tools from clickstream analyses and graph-modeled data clustering with psychometrics. In our approach, we (a) provide similarity measures that are based on both actions and the associated action-level timing data and (b) subsequently employ cluster edge deletion for identifying homogeneous, interpretable, well-separated groups of action patterns, each describing a common response process. Guidelines on how to apply the approach are provided. The approach and its utility are illustrated on a complex problem-solving item from PIAAC 2012.
复杂的互动测试项目在评估中越来越多地被使用。采用互动项目的计算机化评估允许记录时间标记的动作序列。这些序列提供了丰富的信息来源，可以帮助研究考生如何处理项目并得出他们的给定答案。有大量的研究利用动作序列数据来研究考生的行为。然而，相关的计时数据如果有的话，主要也只在项目级别上进行考虑。然而，除了动作序列之外，在动作级别上考虑计时数据，具有极大的潜力来支持对考生行为进行更精细的评估。我们提供了一种方法，该方法联合考虑动作序列和动作级别的时间，以识别常见的反应过程。在这样做的过程中，我们将点击流分析和基于图形模型的数据聚类工具与心理计量学相结合。在我们的方法中，我们（a）提供了基于动作和相关动作级别的计时数据的相似性度量，以及（b）随后采用簇边删除来识别同质的、可解释的、分离良好的动作模式组，每个模式组都描述了一个常见的反应过程。提供了应用该方法的指南。该方法及其效用在 2012 年 PIAAC 的一个复杂的解决问题的项目中得到了说明。
