Gan Lulu, Chen Jing, Wang Luning, Lu Yunfei, Ren Jie
China Table Tennis College, Shanghai University of Sport, Shanghai, China.
Division of Health Professions, Texas A&M University-Texarkana, Texarkana, TX, United States.
Front Psychol. 2025 Jun 30;16:1432711. doi: 10.3389/fpsyg.2025.1432711. eCollection 2025.
This study aims to assess the reliability of subjective evaluations conducted under two information conditions and to explore the influence of observer expertise on the consistency of performance assessments of elite table tennis players.
Observers of varying skill levels were invited to provide subjective evaluations of the elite table tennis players' performance by observing specific rally strokes during the match. A Video Masking Paradigm approach was implemented to conceal motion information during critical moments of scoring and losing. The weighted Kappa coefficient () was employed to evaluate the inter-observer consistency between two observers. The Kendall's coefficient of concordance () is a measure of inter-rater agreement, specifically used for ordinal scales (e.g., Likert five-point scale) when multiple raters are involved.
Intra-observer reliability was good ( = 0.61-0.86), whereas inter-observer consistency between the two observers was low ( = 0.01-0.39). Among the observation indicators, the advanced group showed the lowest consistency in evaluating tactical behavior (without results, = 0.44; with results, = 0.76). Experiment 2: The consistency of the observers in the without results condition (expert group = 0.75 vs. advanced group = 0.57 vs. novice group = 0.66) is lower than in the with results (expert group = 0.84 vs. advanced group = 0.78 vs. novice group = 0.76). Across all three observation indicators, namely stroke quality, tactical intention, and competitive posture, the expert group demonstrated the highest level of consistency, followed by the advanced group, while the novice group exhibited the lowest level of agreement.
Observers with table tennis skill levels demonstrate high intra-observer test-retest reliability in subjective evaluations, but the inter-observer consistency is lower. Different information conditions (with or without results) are key variables affecting the consistency of subjective evaluations. When kinematic information is occluded (without results), the consistency of subjective evaluations decreases. The selection of observation indicators also impacts the consistency of subjective evaluations. Additionally, observers' consistency in subjective evaluations is influenced by their level of experience and skill: the higher the observer's level and experience, the greater the consistency of their subjective evaluations.
本研究旨在评估在两种信息条件下进行的主观评估的可靠性，并探讨观察者专业知识对精英乒乓球运动员表现评估一致性的影响。
邀请不同技能水平的观察者通过观察比赛中特定的连续击球来对精英乒乓球运动员的表现进行主观评估。采用视频掩码范式方法在得分和失分的关键时刻隐藏运动信息。加权卡帕系数（）用于评估两位观察者之间的观察者间一致性。肯德尔和谐系数（）是一种评分者间一致性的度量，特别用于涉及多个评分者的有序量表（例如李克特五点量表）。
观察者内信度良好（=0.61 - 0.86），而两位观察者之间的观察者间一致性较低（=0.01 - 0.39）。在观察指标中，高级组在评估战术行为时一致性最低（无结果时，=0.44；有结果时，=0.76）。实验2：观察者在无结果条件下的一致性（专家组=0.75，高级组=0.57，新手组=0.66）低于有结果时（专家组=0.84，高级组=0.78，新手组=0.76）。在所有三个观察指标，即击球质量、战术意图和竞争姿态方面，专家组表现出最高水平的一致性，其次是高级组，而新手组表现出最低水平的一致性。
具有乒乓球技能水平的观察者在主观评估中表现出较高的观察者内重测信度，但观察者间一致性较低。不同的信息条件（有或无结果）是影响主观评估一致性的关键变量。当运动信息被遮挡（无结果）时，主观评估的一致性会降低。观察指标的选择也会影响主观评估的一致性。此外，观察者在主观评估中的一致性受其经验和技能水平的影响：观察者的水平和经验越高，其主观评估的一致性就越大。
