School of Psychology, Jiangxi Normal University, 99 Ziyang Ave, Nanchang, 330022, Jiangxi, China.
Behav Res Methods. 2023 Apr;55(3):963-980. doi: 10.3758/s13428-021-01779-z. Epub 2022 May 6.
A common observation in ability assessment is that the probability of an examinee giving a correct response drops for end-of-test items due to low motivation, time limits or other factors. On the test-takers' side, this change can be considered performance decline (PD), which can strongly affect test validity and bias respondents' ability estimators. Currently, there is an increasing interest in the detection of PD among researchers and practitioners. Researchers and practitioners found that PD detection fails to achieve acceptable power, which is typically below 0.55. Change-point analysis (CPA), a well-developed statistical method, can be applied to item response sequences to identify whether an abrupt change exists. Existing CPA methods cannot be directly used to detect PD because they are appropriate for two-sided alternative hypotheses. To address these issues, this research firstly develops a CPA method based on Jensen-Shannon divergence to detect PD. Additionally, existing CPA statistics were converted into one-sided statistics to accommodate PD detection. Then, a simulation study was conducted to investigate the performance of the proposed method and compare it with modified CPA statistics. Results show that the proposed CPA method can detect PD with higher power while generating a well-controlled Type-I error rate. Compared against modified CPA statistics, the proposed method exhibits an augmentation in power from 1.0% to 8.2%, with average of 5.7% and higher accuracy in locating the change point. Finally, the proposed method was applied to two real datasets to demonstrate its utility.
在能力评估中，一个常见的观察结果是，由于低动机、时间限制或其他因素，考生在测试结束时答对题的概率会下降。从考生的角度来看，这种变化可以被视为表现下降（PD），这会强烈影响测试的有效性并影响受访者能力估计值的准确性。目前，研究人员和从业者越来越关注 PD 的检测。研究人员和从业者发现，PD 检测无法达到可接受的功效，通常低于 0.55。变化点分析（CPA）是一种成熟的统计方法，可以应用于项目反应序列来确定是否存在突然变化。现有的 CPA 方法不能直接用于检测 PD，因为它们适用于双边备择假设。为了解决这些问题，本研究首先开发了一种基于 Jensen-Shannon 散度的 CPA 方法来检测 PD。此外，将现有的 CPA 统计量转换为单边统计量以适应 PD 检测。然后，进行了一项模拟研究，以调查所提出方法的性能，并将其与修改后的 CPA 统计量进行比较。结果表明，所提出的 CPA 方法可以以更高的功效检测 PD，同时保持良好的控制的Ⅰ类错误率。与修改后的 CPA 统计量相比，所提出的方法的功效提高了 1.0%到 8.2%，平均提高了 5.7%，并且在定位变化点方面更准确。最后，将所提出的方法应用于两个真实数据集，以证明其实用性。
