Department of Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota 55905, USA.
Teach Learn Med. 2009 Jul;21(3):188-94. doi: 10.1080/10401330903014137.
Assessment score reliability is usually based on a single analysis. However, reliability is an essential component of validity and assessment validation and revision is a never-ending cycle. For ongoing assessments over extended time frames, real-time reliability computations may alert users to possible changes in the learning environment that are revealed by variations in reliability over time.
To develop software that calculates the reliability of clinical assessments in real time.
Over 2,400 assessment forms were analyzed. We developed software that calculates reliability in real time. Software accuracy was verified by comparing data from our software with a standard method. Factor analysis determined scale dimensionality.
Correlation between our software and a standard method was excellent (ICC for kappas = 0.97; Cronbach's alphas differed by < 0.03). Cronbach's alpha ranged from 0.94 to 0.97 and weighted kappa ranged from 0.08 to 0.40. Factor analysis confirmed 3 teaching domains.
We describe an accurate method for calculating reliability in real time. The benefit of real time computation is that it provides a mechanism for detecting possible changes (related to curriculum, teachers, and students) in the learning environment indicated by changes in reliability over time. This technique will enable investigators to monitor and detect changes in the reliability of assessment scores and, with future study, isolate aspects of the learning environment that impact on reliability.
评估分数的可靠性通常基于单一分析。然而，可靠性是有效性的一个基本组成部分，评估的验证和修订是一个永无止境的循环。对于在较长时间内进行的持续评估，实时可靠性计算可以提醒用户注意学习环境中可能发生的变化，这些变化是通过可靠性随时间的变化而揭示的。
开发实时计算临床评估可靠性的软件。
分析了超过 2400 份评估表。我们开发了实时计算可靠性的软件。通过将我们的软件数据与标准方法进行比较，验证了软件的准确性。因素分析确定了量表的维度。
我们的软件与标准方法之间的相关性非常好（Kappa 的 ICC = 0.97；Cronbach's alpha 的差异小于 0.03）。Cronbach's alpha 的范围为 0.94 到 0.97，加权 Kappa 的范围为 0.08 到 0.40。因素分析证实了 3 个教学领域。
我们描述了一种准确的实时计算可靠性的方法。实时计算的好处是，它提供了一种机制，可以检测学习环境中可能发生的变化（与课程、教师和学生有关），这些变化是通过可靠性随时间的变化而显示出来的。这种技术将使研究人员能够监测和检测评估分数可靠性的变化，并在未来的研究中，隔离影响可靠性的学习环境的各个方面。
