Li Zhi, Chen Michelle Y, Banerjee Jayanti
Department of Linguistics, College of Arts and Science, University of Saskatchewan, Saskatoon, SK, Canada.
Paragon Testing Enterprises, Vancouver, BC, Canada.
Front Psychol. 2020 Jun 3;11:1088. doi: 10.3389/fpsyg.2020.01088. eCollection 2020.
Addressing differential item functioning (DIF) provides validity evidence to support the interpretation of test scores across groups. Conventional DIF methods flag DIF items statistically, but often fail to consolidate a substantive interpretation. The lack of interpretability of DIF results is particularly pronounced in writing assessment where the matching of test takers' proficiency levels often relies on external variables and the reported DIF effect is frequently small in magnitude. Using responses to a prompt that showed small gender DIF favoring female test takers, we demonstrate a corpus-based approach that helps address DIF interpretation. To provide linguistic insights into the possible sources of the small DIF effect, this study compared a gender-balanced corpus of 826 writing samples matched by test takers' performance on the reading and listening components of the test. Four groups of linguistic features that correspond to the rating dimensions, and thus partially represent the writing construct were analyzed. They include (1) sentiment and social cognition, (2) cohesion, (3) syntactic features, and (4) lexical features. After initial screening, 123 linguistic features, all of which were correlated with the writing scores, were retained for gender comparison. Among these selected features, female test takers' writing samples scored higher on six of them with small effect sizes in the categories of cohesion and syntactic features. Three of the six features were positively correlated with higher writing scores, while the other three were negative. These results are largely consistent with previous findings of gender differences in language use. Additionally, the small differences in the language features of the writing samples (in terms of the small number of features that differ between genders and the small effect size of the observed differences) are consistent with the previous DIF results, both suggesting that the effect of gender differences on the writing scores is likely to be very small. In sum, the corpus-based findings provide linguistic insights into the gender-related language differences and their potential consequences in a testing context. These findings are meaningful for furthering our understanding of the small gender DIF effect identified through statistical analysis, which lends support to the validity of writing scores.
处理项目功能差异（DIF）可为跨群体测试分数的解释提供效度证据。传统的DIF方法通过统计手段标记出存在DIF的项目，但往往无法给出实质性的解释。DIF结果缺乏可解释性在写作评估中尤为明显，因为考生熟练程度的匹配通常依赖外部变量，而且报告的DIF效应往往幅度较小。通过对一道显示出有利于女性考生的微小性别DIF的题目作答情况，我们展示了一种基于语料库的方法，有助于解决DIF解释问题。为了从语言角度深入了解微小DIF效应的可能来源，本研究比较了一个由826篇写作样本组成的性别均衡语料库，这些样本在考生的阅读和听力部分成绩上进行了匹配。分析了与评分维度相对应、从而部分代表写作结构的四组语言特征。它们包括：（1）情感与社会认知，（2）衔接，（3）句法特征，以及（4）词汇特征。经过初步筛选，保留了123个与写作分数相关的语言特征用于性别比较。在这些选定的特征中，女性考生的写作样本在其中六个特征上得分更高，在衔接和句法特征类别中效应量较小。这六个特征中有三个与较高的写作分数呈正相关，另外三个呈负相关。这些结果在很大程度上与先前关于语言使用中性别差异的研究结果一致。此外，写作样本语言特征的微小差异（就性别之间存在差异的特征数量少以及观察到的差异效应量小而言）与先前的DIF结果一致，两者都表明性别差异对写作分数的影响可能非常小。总之，基于语料库的研究结果为与性别相关的语言差异及其在测试情境中的潜在影响提供了语言方面的见解。这些发现对于深化我们对通过统计分析确定的微小性别DIF效应的理解具有重要意义，这为写作分数的效度提供了支持。
