Department of Biomedical Engineering, Tsinghua University, Beijing, People's Republic of China.
J Neural Eng. 2013 Apr;10(2):026014. doi: 10.1088/1741-2560/10/2/026014. Epub 2013 Feb 28.
Today, the brain-computer interface (BCI) community lacks a standard method to evaluate an online BCI's performance. Even the most commonly used metric, the information transfer rate (ITR), is often reported differently, even incorrectly, in many papers, which is not conducive to BCI research. This paper aims to point out many of the existing problems and give some suggestions and methods to overcome these problems.
First, the preconditions inherent in ITR calculation based on Wolpaw's definition are summarized and several incorrect ITR calculations, which go against the preconditions, are indicated. Then, the issues affecting ITR estimation during the test of online BCI systems are discussed in detail. Finally, a task-oriented online BCI test platform was proposed, which may help BCI evaluations in real-world applications.
The guidelines for ITR calculation in online BCIs testing are proposed. The platform executed in the Beijing BCI Competition 2010 shows that it can be used as a common way to compare the online performances (including the ITR) of existing BCI paradigms.
The proposed guidelines and task-oriented test platform may reduce the uncertainty and artifacts of online BCI performance evaluation; they provide a relatively objective way to compare different BCI's performances in real-world BCI applications, which is a forward step toward developing standards for BCI performance evaluation.
目前，脑机接口（BCI）领域缺乏评估在线 BCI 性能的标准方法。即使是最常用的指标，信息传输率（ITR），在许多论文中也经常以不同的方式，甚至不正确地报告，这不利于 BCI 研究。本文旨在指出许多现有的问题，并提出一些建议和方法来克服这些问题。
首先总结了基于 Wolpaw 定义的 ITR 计算所固有的前提条件，并指出了几个违反前提条件的不正确的 ITR 计算。然后，详细讨论了在线 BCI 系统测试中影响 ITR 估计的问题。最后，提出了一个面向任务的在线 BCI 测试平台，这可能有助于真实应用中的 BCI 评估。
提出了在线 BCI 测试中 ITR 计算的指南。在北京脑机接口竞赛 2010 中执行的平台表明，它可以作为比较现有 BCI 范式的在线性能（包括 ITR）的常用方法。
提出的指南和面向任务的测试平台可以减少在线 BCI 性能评估的不确定性和伪影；它们为在真实的 BCI 应用中比较不同 BCI 的性能提供了一种相对客观的方法，这是朝着开发 BCI 性能评估标准迈出的一步。
