Department of Psychology, Carnegie Mellon, United States.
Department of Psychology, Carnegie Mellon, United States.
Neuroimage. 2020 Nov 1;221:116999. doi: 10.1016/j.neuroimage.2020.116999. Epub 2020 Jun 1.
We describe the Sketch-and-Stitch method for bringing together a cognitive model and EEG to reconstruct the cognition of a subject. The method was tested in the context of a video game where the actions are highly interdependent and variable: simply changing whether a key was pressed or not for a 30th of a second can lead to a very different outcome. The Sketch level identifies the critical events in the game and the Stitch level fills in the detailed actions between these events. The critical events tend to produce robust EEG signals and the cognitive model provides probabilities of various transitions between critical events and the distribution of intervals between these events. This information can be combined in a hidden semi-Markov model that identifies the most probable sequence of critical events and when they happened. The Stitch level selects detailed actions from an extensive library of model games to produce these critical events. The decision about which sequence of actions to select from the library is made on the basis of how well they would produce weaker aspects of the EEG signal. The resulting approach can produce quite compelling replays of actual games from the EEG of a subject.
我们描述了 Sketch-and-Stitch 方法，用于将认知模型和 EEG 结合起来，以重建受试者的认知。该方法在视频游戏的背景下进行了测试，其中的动作高度相互依赖且多变：仅仅改变按键的按下或不按下状态持续 30 分之一秒，就可能导致非常不同的结果。Sketch 级别确定游戏中的关键事件，而 Stitch 级别则在这些事件之间填充详细的动作。关键事件往往会产生稳健的 EEG 信号，而认知模型则提供了各种关键事件之间的转换概率以及这些事件之间的间隔分布。这些信息可以在隐式半马尔可夫模型中进行组合，以识别最可能的关键事件序列及其发生时间。Stitch 级别从广泛的模型游戏库中选择详细的动作来产生这些关键事件。从库中选择动作序列的决策是基于它们对 EEG 信号较弱方面的产生效果来做出的。由此产生的方法可以根据受试者的 EEG 生成实际游戏的非常引人入胜的重播。
