Monseur Christian, Berezner Alla
Universite de Liege, FAPSE, Department Education, Bld. du Rectorat, 5 (B32) 4000 Liege, Belgium.
J Appl Meas. 2007;8(3):323-35.
Since the IEA's Third International Mathematics and Science Study, one of the major objectives of international surveys in education has been to report trends in achievement. The names of the two current IEA surveys reflect this growing interest: Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS). Similarly a central concern of the OECD's PISA is with trends in outcomes over time. To facilitate trend analyses these studies link their tests using common item equating in conjunction with item response modelling methods. IEA and PISA policies differ in terms of reporting the error associated with trends. In IEA surveys, the standard errors of the trend estimates do not include the uncertainty associated with the linking step while PISA does include a linking error component in the standard errors of trend estimates. In other words, PISA implicitly acknowledges that trend estimates partly depend on the selected common items, while the IEA's surveys do not recognise this source of error. Failing to recognise the linking error leads to an underestimation of the standard errors and thus increases the Type I error rate, thereby resulting in reporting of significant changes in achievement when in fact these are not significant. The growing interest of policy makers in trend indicators and the impact of the evaluation of educational reforms appear to be incompatible with such underestimation. However, the procedure implemented by PISA raises a few issues about the underlying assumptions for the computation of the equating error. After a brief introduction, this paper will describe the procedure PISA implemented to compute the linking error. The underlying assumptions of this procedure will then be discussed. Finally an alternative method based on replication techniques will be presented, based on a simulation study and then applied to the PISA 2000 data.
自国际教育成就评价协会（IEA）的第三次国际数学和科学研究以来，国际教育调查的主要目标之一就是报告成绩趋势。IEA当前两项调查的名称反映了这种日益增长的兴趣：国际数学和科学趋势研究（TIMSS）以及国际阅读素养进展研究（PIRLS）。同样，经济合作与发展组织（OECD）的国际学生评估项目（PISA）的一个核心关注点也是随着时间推移的成绩趋势。为便于进行趋势分析，这些研究通过使用共同项目等值结合项目反应建模方法来关联其测试。IEA和PISA在报告与趋势相关的误差方面政策有所不同。在IEA调查中，趋势估计的标准误差不包括与等值步骤相关的不确定性，而PISA在趋势估计的标准误差中确实包含了一个等值误差成分。换句话说，PISA含蓄地承认趋势估计部分取决于所选的共同项目，而IEA的调查没有认识到这一误差来源。未能认识到等值误差会导致对标准误差的低估，从而增加I类错误率，进而导致在实际上成绩变化不显著时却报告为显著变化。政策制定者对趋势指标的兴趣日益浓厚以及教育改革评估的影响似乎与这种低估不相容。然而，PISA实施的程序引发了一些关于等值误差计算的基本假设的问题。在简要介绍之后，本文将描述PISA为计算等值误差所实施的程序。然后将讨论该程序的基本假设。最后，将基于一项模拟研究并应用于PISA 2000数据，提出一种基于重复技术的替代方法。
