Blau Ari, Schaffer Evan S, Mishra Neeli, Miska Nathaniel J, Paninski Liam, Whiteway Matthew R
Department of Statistics, Columbia University.
Icahn School of Medicine, Mount Sinai.
ArXiv. 2024 Dec 17:arXiv:2407.16727v2.
Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms - which include tree-based models, deep neural networks, and graphical models - differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species - fly, mouse, and human - we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.
行为视频的动作分割是将每个帧标记为属于一个或多个离散类别的过程，并且是许多研究动物行为的研究的关键组成部分。存在各种各样的算法来自动解析离散的动物行为，包括监督学习、无监督学习和半监督学习范式。这些算法——包括基于树的模型、深度神经网络和图形模型——在结构和对数据的假设上有很大差异。我们使用跨越多个物种（果蝇、小鼠和人类）的四个数据集，系统地研究了这些不同算法的输出如何与感兴趣的手动注释行为对齐。在此过程中，我们引入了一个半监督动作分割模型，该模型弥合了监督深度神经网络和无监督图形模型之间的差距。我们发现，在观测中添加时间信息的全监督时间卷积网络在我们所有数据集的监督指标上表现最佳。
