Cetingül H Ertan, Yemez Yücel, Erzin Engin, Tekalp A Murat
Multimedia, Vision and Graphics Laboratory, College of Engineering, Koç University, Sariyer, Istanbul, Turkey.
IEEE Trans Image Process. 2006 Oct;15(10):2879-91. doi: 10.1109/tip.2006.877528.
There have been several studies that jointly use audio, lip intensity, and lip geometry information for speaker identification and speech-reading applications. This paper proposes using explicit lip motion information, instead of or in addition to lip intensity and/or geometry information, for speaker identification and speech-reading within a unified feature selection and discrimination analysis framework, and addresses two important issues: 1) Is using explicit lip motion information useful, and, 2) if so, what are the best lip motion features for these two applications? The best lip motion features for speaker identification are considered to be those that result in the highest discrimination of individual speakers in a population, whereas for speech-reading, the best features are those providing the highest phoneme/word/phrase recognition rate. Several lip motion feature candidates have been considered including dense motion features within a bounding box about the lip, lip contour motion features, and combination of these with lip shape features. Furthermore, a novel two-stage, spatial, and temporal discrimination analysis is introduced to select the best lip motion features for speaker identification and speech-reading applications. Experimental results using an hidden-Markov-model-based recognition system indicate that using explicit lip motion information provides additional performance gains in both applications, and lip motion features prove more valuable in the case of speech-reading application.
已有多项研究联合使用音频、唇部强度和唇部几何信息用于说话人识别和语音阅读应用。本文提出在统一的特征选择和判别分析框架内，使用明确的唇部运动信息，而非唇部强度和/或几何信息，或作为其补充，用于说话人识别和语音阅读，并解决两个重要问题：1）使用明确的唇部运动信息是否有用，以及2）如果有用，对于这两种应用而言，最佳的唇部运动特征是什么？对于说话人识别，最佳的唇部运动特征被认为是那些能在人群中对个体说话人产生最高区分度的特征，而对于语音阅读，最佳特征是那些能提供最高音素/单词/短语识别率的特征。已考虑了多个唇部运动特征候选，包括围绕唇部的边界框内的密集运动特征、唇部轮廓运动特征，以及这些特征与唇部形状特征的组合。此外，还引入了一种新颖的两阶段、空间和时间判别分析，以选择用于说话人识别和语音阅读应用的最佳唇部运动特征。使用基于隐马尔可夫模型的识别系统的实验结果表明，使用明确的唇部运动信息在这两种应用中均能带来额外的性能提升，并且在语音阅读应用中，唇部运动特征被证明更具价值。
