Spring Aaron M, Pittman Daniel J, Rizwan Arsalan, Aghakhani Yahya, Jirsch Jeffrey, Connolly Mary, Wiebe Samuel, Appendino Juan Pablo, Datta Anita, Steve Trevor, Pillay Neelan, Javidan Manouchehr, Scantlebury Morris, Hrazdil Chantelle, Josephson Colin Bruce, Boelman Cyrus, Gross Donald, Singh Shaily, Bello-Espinosa Luis, Huh Linda, Jetté Nathalie, Federico Paolo
Department of Clinical Neurosciences, University of Calgary, Calgary, AB, Canada.
Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada.
Front Neurol. 2022 Feb 14;13:794668. doi: 10.3389/fneur.2022.794668. eCollection 2022.
We examined the effect of a simple Delphi-method feedback on visual identification of high frequency oscillations (HFOs) in the ripple (80-250 Hz) band, and assessed the impact of this training intervention on the interrater reliability and generalizability of HFO evaluations.
We employed a morphology detector to identify potential HFOs at two thresholds and presented them to visual reviewers to assess the probability of each epoch containing an HFO. We recruited 19 board-certified epileptologists with various levels of experience to complete a series of HFO evaluations during three sessions. A Delphi-style intervention was used to provide feedback on the performance of each reviewer relative to their peers. A delayed-intervention paradigm was used, in which reviewers received feedback either before or after the second session. ANOVAs were used to assess the effect of the intervention on the reviewers' evaluations. Generalizability theory was used to assess the interrater reliability before and after the intervention.
The intervention, regardless of when it occurred, resulted in a significant reduction in the variability between reviewers in both groups ( = 0.037, = 0.003). Prior to the delayed-intervention, the group receiving the early intervention showed a significant reduction in variability ( = 0.041), but the delayed-intervention group did not ( = 0.414). Following the intervention, the projected number of reviewers required to achieve strong generalizability decreased from 35 to 16.
This study shows a robust effect of a Delphi-style intervention on the interrater variability, reliability, and generalizability of HFO evaluations. The observed decreases in HFO marking discrepancies across 14 of the 15 reviewers are encouraging: they are necessarily associated with an increase in interrater reliability, and therefore with a corresponding decrease in the number of reviewers required to achieve strong generalizability. Indeed, the reliability of all reviewers following the intervention was similar to that of experienced reviewers prior to intervention. Therefore, a Delphi-style intervention could be implemented either to sufficiently train any reviewer, or to further refine the interrater reliability of experienced reviewers. In either case, a Delphi-style intervention would help facilitate the standardization of HFO evaluations and its implementation in clinical care.
我们研究了一种简单的德尔菲法反馈对视觉识别涟漪频段（80 - 250Hz）高频振荡（HFOs）的影响，并评估了这种训练干预对HFO评估者间可靠性和可推广性的影响。
我们使用一种形态检测器在两个阈值下识别潜在的HFOs，并将其呈现给视觉审查者，以评估每个时段包含HFO的概率。我们招募了19名具有不同经验水平的获得董事会认证的癫痫专家，在三个阶段完成一系列HFO评估。采用德尔菲式干预，以提供每位审查者相对于其同行表现的反馈。使用延迟干预范式，其中审查者在第二阶段之前或之后收到反馈。使用方差分析来评估干预对审查者评估的影响。使用可推广性理论来评估干预前后评估者间的可靠性。
无论干预何时发生，两组审查者之间的变异性均显著降低（F = 0.037，p = 0.003）。在延迟干预之前，接受早期干预的组变异性显著降低（F = 0.041），但延迟干预组没有（F = 0.414）。干预后，实现强可推广性所需的审查者预计数量从35人减少到16人。
本研究表明德尔菲式干预对HFO评估的评估者间变异性、可靠性和可推广性有显著影响。在15名审查者中的14名中观察到的HFO标记差异减少令人鼓舞：它们必然与评估者间可靠性的提高相关，因此与实现强可推广性所需的审查者数量相应减少相关。事实上，干预后所有审查者的可靠性与干预前经验丰富的审查者相似。因此，可以实施德尔菲式干预来充分培训任何审查者或进一步提高经验丰富的审查者的评估者间可靠性。在任何一种情况下，德尔菲式干预都将有助于促进HFO评估的标准化及其在临床护理中的应用。
