DIPF | Leibniz Institute for Research and Information in Education, Frankfurt am Main, Germany.
Centre for International Student Assessment (ZIB), Frankfurt am Main, Germany.
Br J Educ Psychol. 2019 Sep;89(3):524-537. doi: 10.1111/bjep.12278. Epub 2019 Apr 13.
With digital technologies, competence assessments can provide process data, such as mouse clicks with corresponding timestamps, as additional information about the skills and strategies of test takers. However, in order to use variables generated from process data sensibly for educational purposes, their interpretation needs to be validated with regard to their intended meaning.
This study seeks to demonstrate how process data from an assessment of multiple document comprehension can be used to represent sourcing, which summarizes activities for the consideration of the origin and intention of documents. The investigated process variables were created according to theoretical assumptions about sourcing, and systematically tested for differences between persons, units (i.e., documents and items), and properties of the test administration.
The sample included 310 German university students (79.4% female), enrolled in several bachelor's or master's programmes of the social sciences and humanities.
Regarding the hierarchical data structure, the hypotheses were analysed with generalized linear mixed models (GLMM).
The results mostly revealed expected differences between individuals and units. However, unexpected effects of the administered order of units and documents were detected.
The study demonstrates the theory-informed construction of process variables from log-files and an approach for empirical validation of their interpretation. The results suggest that students apply sourcing for different reasons, but also stress the need of further validation studies and refinements in the operationalization of the indicators investigated.
随着数字技术的发展，能力评估可以提供过程数据，例如带有相应时间戳的鼠标点击，作为测试者技能和策略的附加信息。然而，为了明智地将从过程数据中生成的变量用于教育目的，需要根据其预期含义验证其解释。
本研究旨在展示如何使用多项文档理解评估的过程数据来表示源分析，该分析总结了考虑文档来源和意图的活动。所研究的过程变量是根据关于源分析的理论假设创建的，并系统地测试了个体、单位（即文档和项目）以及测试管理的属性之间的差异。
样本包括 310 名德国大学生（女性占 79.4%），他们就读于社会科学和人文学科的多个本科或硕士课程。
关于分层数据结构，使用广义线性混合模型（GLMM）分析了假设。
结果主要揭示了个体和单位之间的预期差异。然而，检测到了单位和文档的管理顺序的意外影响。
本研究展示了从日志文件构建过程变量的理论指导，并提出了一种验证其解释的实证方法。结果表明，学生出于不同的原因应用源分析，但也强调了进一步验证研究和所研究指标的操作细化的必要性。
