Tsipas Nikolaos, Vrysis Lazaros, Konstantoudakis Konstantinos, Dimoulas Charalampos
Aristotle University of Thessaloniki, Thessaloniki, Greece.
J Acoust Soc Am. 2020 Dec;148(6):3751. doi: 10.1121/10.0002924.
In this paper, an audio-driven, multimodal approach for speaker diarization in multimedia content is introduced and evaluated. The proposed algorithm is based on semi-supervised clustering of audio-visual embeddings, generated using deep learning techniques. The two modes, audio and video, are separately addressed; a long short-term memory Siamese neural network is employed to produce embeddings from audio, whereas a pre-trained convolutional neural network is deployed to generate embeddings from two-dimensional blocks representing the faces of speakers detected in video frames. In both cases, the models are trained using cost functions that favor smaller spatial distances between samples from the same speaker and greater spatial distances between samples from different speakers. A fusion stage, based on hypotheses derived from the established practices in television content production, is deployed on top of the unimodal sub-components to improve speaker diarization performance. The proposed methodology is evaluated against VoxCeleb, a large-scale dataset with hundreds of available speakers and AVL-SD, a newly developed, publicly available dataset aiming at capturing the peculiarities of TV news content under different scenarios. In order to promote reproducible research and collaboration in the field, the implemented algorithm is provided as an open-source software package.
本文介绍并评估了一种用于多媒体内容中说话人聚类的音频驱动多模态方法。所提出的算法基于使用深度学习技术生成的视听嵌入的半监督聚类。音频和视频这两种模态被分别处理；使用长短期记忆连体神经网络从音频中生成嵌入，而部署一个预训练的卷积神经网络从表示视频帧中检测到的说话人面部的二维块中生成嵌入。在这两种情况下，模型使用代价函数进行训练，该代价函数有利于同一说话人的样本之间具有较小的空间距离，以及不同说话人的样本之间具有较大的空间距离。基于电视内容制作中的既定做法得出的假设，在单模态子组件之上部署了一个融合阶段，以提高说话人聚类性能。所提出的方法针对VoxCeleb（一个拥有数百个可用说话人的大规模数据集）和AVL-SD（一个新开发的、公开可用的数据集，旨在捕捉不同场景下电视新闻内容的特点）进行评估。为了促进该领域的可重复研究和合作，所实现的算法作为一个开源软件包提供。
