Wolfe Katie, Seaman Michael A, Drasgow Erik
University of South Carolina, Columbia, USA
University of South Carolina, Columbia, USA.
Behav Modif. 2016 Nov;40(6):852-873. doi: 10.1177/0145445516644699. Epub 2016 Apr 21.
Previous research on visual analysis has reported low levels of interrater agreement. However, many of these studies have methodological limitations (e.g., use of AB designs, undefined judgment task) that may have negatively influenced agreement. Our primary purpose was to evaluate whether agreement would be higher than previously reported if we addressed these weaknesses. Our secondary purposes were to investigate agreement at the tier level (i.e., the AB comparison) and at the functional relation level in multiple baseline designs and to examine the relationship between raters' decisions at each of these levels. We asked experts (N = 52) to make judgments about changes in the dependent variable in individual tiers and about the presence of an overall functional relation in 31 multiple baseline graphs. Our results indicate that interrater agreement was just at or just below minimally adequate levels for both types of decisions and that agreement at the individual tier level often resulted in agreement about the overall functional relation. We report additional findings and discuss implications for practice and future research.
先前关于视觉分析的研究报告称评分者间的一致性水平较低。然而，这些研究中有许多存在方法学上的局限性（例如，采用AB设计、未明确判断任务），这可能对一致性产生了负面影响。我们的主要目的是评估如果我们解决这些弱点，一致性是否会高于先前报告的水平。我们的次要目的是在多个基线设计中研究层级水平（即AB比较）和功能关系水平上的一致性，并检验评分者在每个这些水平上的决策之间的关系。我们让52位专家对31个多个基线图表中各个层级的因变量变化以及整体功能关系的存在情况进行判断。我们的结果表明，对于这两种类型的决策，评分者间的一致性刚好处于或略低于最低可接受水平，并且在个体层级水平上的一致性常常导致对整体功能关系的一致判断。我们报告了其他发现，并讨论了对实践和未来研究的启示。
