Forthmann Boris, Förster Natalie, Schütze Birgit, Hebbecker Karin, Flessner Janis, Peters Martin T, Souvignier Elmar
Institute of Psychology in Education, University of Münster, 48149 Münster, Germany.
J Intell. 2020 Mar 9;8(1):11. doi: 10.3390/jintelligence8010011.
Distractors might display discriminatory power with respect to the construct of interest (e.g., intelligence), which was shown in recent applications of nested logit models to the short-form of Raven's progressive matrices and other reasoning tests. In this vein, a simulation study was carried out to examine two effect size measures (i.e., a variant of Cohen's ω and the canonical correlation ) for their potential to detect distractors with ability-related discriminatory power. The simulation design was adopted to item selection scenarios relying on rather small sample sizes (e.g., = 100 or = 200). Both suggested effect size measures (Cohen's ω only when based on two ability groups) yielded acceptable to conservative type-I-error rates, whereas, the canonical correlation outperformed Cohen's ω in terms of empirical power. The simulation results further suggest that an effect size threshold of 0.30 is more appropriate as compared to more lenient (0.10) or stricter thresholds (0.50). The suggested item-analysis procedure is illustrated with an analysis of twelve Raven's progressive matrices items in a sample of = 499 participants. Finally, strategies for item selection for cognitive ability tests with the goal of scaling by means of nested logit models are discussed.
干扰项可能会在感兴趣的结构（如智力）方面表现出区分能力，这在最近将嵌套逻辑模型应用于瑞文渐进矩阵简版及其他推理测试的研究中得到了体现。在此背景下，开展了一项模拟研究，以检验两种效应量指标（即科恩ω的一种变体和典型相关）检测具有与能力相关区分能力的干扰项的潜力。模拟设计适用于样本量较小（如N = 100或N = 200）的项目选择场景。两种建议的效应量指标（仅基于两个能力组时的科恩ω）产生了可接受至保守的I类错误率，而典型相关在实证功效方面优于科恩ω。模拟结果还表明，与更宽松的阈值（0.10）或更严格的阈值（0.50）相比，效应量阈值0.30更为合适。通过对499名参与者样本中的12个瑞文渐进矩阵项目进行分析，展示了所建议的项目分析程序。最后，讨论了以嵌套逻辑模型进行量表编制为目标的认知能力测试项目选择策略。
