Huang Shan, Ishii Hidetoki
Nagoya University, Japan.
Appl Psychol Meas. 2024 Dec 19:01466216241310602. doi: 10.1177/01466216241310602.
Many studies on differential item functioning (DIF) detection rely on single detection methods (SDMs), each of which necessitates specific assumptions that may not always be validated. Using an inappropriate SDM can lead to diminished accuracy in DIF detection. To address this limitation, a novel multi-detector combination (MDC) approach is proposed. Unlike SDMs, MDC effectively evaluates the relevance of different SDMs under various test conditions and integrates them using supervised learning, thereby mitigating the risk associated with selecting a suboptimal SDM for DIF detection. This study aimed to validate the accuracy of the MDC approach by applying five types of SDMs and four distinct supervised learning methods in MDC modeling. Model performance was assessed using the area under the curve (AUC), which provided a comprehensive measure of the ability of the model to distinguish between classes across all threshold levels, with higher AUC values indicating higher accuracy. The MDC methods consistently achieved higher average AUC values compared to SDMs in both matched test sets (where test conditions align with the training set) and unmatched test sets. Furthermore, MDC outperformed all SDMs under each test condition. These findings indicated that MDC is highly accurate and robust across diverse test conditions, establishing it as a viable method for practical DIF detection.
许多关于差异项目功能（DIF）检测的研究依赖于单一检测方法（SDM），每种方法都需要特定的假设，而这些假设可能并不总是成立。使用不适当的SDM可能会导致DIF检测的准确性降低。为了解决这一局限性，提出了一种新颖的多检测器组合（MDC）方法。与SDM不同，MDC能在各种测试条件下有效评估不同SDM的相关性，并使用监督学习将它们整合起来，从而降低为DIF检测选择次优SDM所带来的风险。本研究旨在通过在MDC建模中应用五种类型的SDM和四种不同的监督学习方法来验证MDC方法的准确性。使用曲线下面积（AUC）评估模型性能，AUC全面衡量了模型在所有阈值水平上区分不同类别的能力，AUC值越高表明准确性越高。在匹配测试集（测试条件与训练集一致）和不匹配测试集中，MDC方法的平均AUC值始终高于SDM。此外，在每种测试条件下，MDC的表现均优于所有SDM。这些发现表明，MDC在各种不同的测试条件下都具有很高的准确性和稳健性，使其成为实际DIF检测的可行方法。
