Department of Biomedical Engineering, University of Miami, Coral Gables, FL, United States of America.
J Neural Eng. 2018 Jun;15(3):036009. doi: 10.1088/1741-2552/aa9dbe. Epub 2017 Nov 28.
Conventional decoding pipeline for brain-machine interfaces (BMIs) consists of chained different stages of feature extraction, time-frequency analysis and statistical learning models. Each of these stages uses a different algorithm trained in a sequential manner, which makes it difficult to make the whole system adaptive. The goal was to create an adaptive online system with a single objective function and a single learning algorithm so that the whole system can be trained in parallel to increase the decoding performance. Here, we used deep neural networks consisting of convolutional neural networks (CNN) and a special kind of recurrent neural network (RNN) called long short term memory (LSTM) to address these needs.
We used electrocorticography (ECoG) data collected by Kubanek et al. The task consisted of individual finger flexions upon a visual cue. Our model combined a hierarchical feature extractor CNN and a RNN that was able to process sequential data and recognize temporal dynamics in the neural data. CNN was used as the feature extractor and LSTM was used as the regression algorithm to capture the temporal dynamics of the signal.
We predicted the finger trajectory using ECoG signals and compared results for the least angle regression (LARS), CNN-LSTM, random forest, LSTM model (LSTM_HC, for using hard-coded features) and a decoding pipeline consisting of band-pass filtering, energy extraction, feature selection and linear regression. The results showed that the deep learning models performed better than the commonly used linear model. The deep learning models not only gave smoother and more realistic trajectories but also learned the transition between movement and rest state.
This study demonstrated a decoding network for BMI that involved a convolutional and recurrent neural network model. It integrated the feature extraction pipeline into the convolution and pooling layer and used LSTM layer to capture the state transitions. The discussed network eliminated the need to separately train the model at each step in the decoding pipeline. The whole system can be jointly optimized using stochastic gradient descent and is capable of online learning.
传统的脑机接口（BMI）解码流水线由特征提取、时频分析和统计学习模型的不同阶段组成。这些阶段中的每一个都使用不同的算法进行顺序训练，这使得整个系统很难自适应。目标是创建一个具有单一目标函数和单一学习算法的自适应在线系统，以便整个系统可以并行训练，以提高解码性能。在这里，我们使用由卷积神经网络（CNN）和一种特殊的递归神经网络（RNN）组成的深度神经网络，即长短期记忆（LSTM）来满足这些需求。
我们使用了由库巴内克等人收集的皮层电图（ECoG）数据。任务是在视觉提示下进行单个手指弯曲。我们的模型结合了分层特征提取 CNN 和能够处理顺序数据并识别神经数据中时间动态的 RNN。CNN 用作特征提取器，LSTM 用作回归算法，以捕获信号的时间动态。
我们使用 ECoG 信号预测手指轨迹，并将结果与最小角度回归（LARS）、CNN-LSTM、随机森林、LSTM 模型（LSTM_HC，用于使用硬编码特征）和由带通滤波、能量提取、特征选择和线性回归组成的解码流水线进行比较。结果表明，深度学习模型的性能优于常用的线性模型。深度学习模型不仅给出了更平滑和更现实的轨迹，而且还学习了运动和静止状态之间的转换。
这项研究展示了一种涉及卷积和递归神经网络模型的 BMI 解码网络。它将特征提取管道集成到卷积和池化层中，并使用 LSTM 层来捕获状态转换。所讨论的网络消除了在解码流水线的每个步骤中分别训练模型的需要。整个系统可以使用随机梯度下降联合优化，并具有在线学习能力。
