Department of Psychology, University of Konstanz Konstanz, Germany ; Zukunftskolleg, University of Konstanz Konstanz, Germany ; Department of Society and Economics, Rhine-Waal University of Applied Sciences Kleve, Germany.
Department of Psychology, University of Konstanz Konstanz, Germany ; Zukunftskolleg, University of Konstanz Konstanz, Germany.
Front Psychol. 2014 Jun 4;5:509. doi: 10.3389/fpsyg.2014.00509. eCollection 2014.
This report has two main purposes. First, we combine well-known analytical approaches to conduct a comprehensive assessment of agreement and correlation of rating-pairs and to dis-entangle these often confused concepts, providing a best-practice example on concrete data and a tutorial for future reference. Second, we explore whether a screening questionnaire developed for use with parents can be reliably employed with daycare teachers when assessing early expressive vocabulary. A total of 53 vocabulary rating pairs (34 parent-teacher and 19 mother-father pairs) collected for two-year-old children (12 bilingual) are evaluated. First, inter-rater reliability both within and across subgroups is assessed using the intra-class correlation coefficient (ICC). Next, based on this analysis of reliability and on the test-retest reliability of the employed tool, inter-rater agreement is analyzed, magnitude and direction of rating differences are considered. Finally, Pearson correlation coefficients of standardized vocabulary scores are calculated and compared across subgroups. The results underline the necessity to distinguish between reliability measures, agreement and correlation. They also demonstrate the impact of the employed reliability on agreement evaluations. This study provides evidence that parent-teacher ratings of children's early vocabulary can achieve agreement and correlation comparable to those of mother-father ratings on the assessed vocabulary scale. Bilingualism of the evaluated child decreased the likelihood of raters' agreement. We conclude that future reports of agreement, correlation and reliability of ratings will benefit from better definition of terms and stricter methodological approaches. The methodological tutorial provided here holds the potential to increase comparability across empirical reports and can help improve research practices and knowledge transfer to educational and therapeutic settings.
本报告有两个主要目的。首先，我们结合了著名的分析方法，对评分对的一致性和相关性进行了全面评估，并对这两个经常混淆的概念进行了区分，为具体数据提供了最佳实践示例，并为未来的参考提供了教程。其次，我们探讨了为家长开发的筛选问卷是否可以在评估早期表达性词汇时，可靠地用于日托教师。共评估了 53 对词汇评分（34 对家长-教师和 19 对母亲-父亲），这些评分是为两岁儿童（12 名双语儿童）收集的。首先，使用组内相关系数（ICC）评估了亚组内和亚组间的评分者间可靠性。接下来，基于对可靠性的分析以及所使用工具的测试-重测可靠性，分析了评分者间的一致性，考虑了评分差异的大小和方向。最后，计算了标准化词汇得分的皮尔逊相关系数，并在亚组间进行了比较。结果强调了区分可靠性度量、一致性和相关性的必要性。它们还表明了所使用可靠性对一致性评估的影响。本研究表明，在评估的词汇量表上，儿童早期词汇的家长-教师评分可以达到与母亲-父亲评分相当的一致性和相关性。被评估儿童的双语能力降低了评分者达成一致的可能性。我们得出结论，未来的评分一致性、相关性和可靠性报告将受益于术语的更好定义和更严格的方法学方法。这里提供的方法学教程有可能提高跨实证报告的可比性，并有助于改善研究实践和知识转移到教育和治疗环境中。
