Chen Zhanli, Ansari Rashid, Wilkie Diana J
Department of Electrical and Computer Engineering, University of Illinois at Chicago.
Department of Biobehavioral Nursing, University of Florida.
IEEE Trans Affect Comput. 2022 Jan-Mar;13(1):135-146. doi: 10.1109/taffc.2019.2949314. Epub 2019 Oct 30.
Patient pain can be detected highly reliably from facial expressions using a set of facial muscle-based action units (AUs) defined by the Facial Action Coding System (FACS). A key characteristic of facial expression of pain is the simultaneous occurrence of pain-related AU combinations, whose automated detection would be highly beneficial for efficient and practical pain monitoring. Existing general Automated Facial Expression Recognition (AFER) systems prove inadequate when applied specifically for detecting pain as they either focus on detecting individual pain-related AUs but not on combinations or they seek to bypass AU detection by training a binary pain classifier directly on pain intensity data but are limited by lack of enough labeled data for satisfactory training. In this paper, we propose a new approach that mimics the strategy of human coders of decoupling pain detection into two consecutive tasks: one performed at the individual video-frame level and the other at video-sequence level. Using state-of-the-art AFER tools to detect single AUs at the frame level, we propose two novel data structures to encode AU combinations from single AU scores. Two weakly supervised learning frameworks namely multiple instance learning (MIL) and multiple clustered instance learning (MCIL) are employed corresponding to each data structure to learn pain from video sequences. Experimental results show an 87% pain recognition accuracy with 0.94 AUC (Area Under Curve) on the UNBC-McMaster Shoulder Pain Expression dataset. Tests on long videos in a lung cancer patient video dataset demonstrates the potential value of the proposed system for pain monitoring in clinical settings.
使用由面部动作编码系统（FACS）定义的一组基于面部肌肉的动作单元（AU），可以从面部表情中高度可靠地检测出患者的疼痛。疼痛面部表情的一个关键特征是与疼痛相关的AU组合同时出现，其自动检测对于高效且实用的疼痛监测将非常有益。现有的通用自动面部表情识别（AFER）系统在专门用于检测疼痛时被证明是不够的，因为它们要么专注于检测单个与疼痛相关的AU，而不是组合，要么试图通过直接在疼痛强度数据上训练二元疼痛分类器来绕过AU检测，但受到缺乏足够标记数据进行满意训练的限制。在本文中，我们提出了一种新方法，该方法模仿人类编码员将疼痛检测解耦为两个连续任务的策略：一个在单个视频帧级别执行，另一个在视频序列级别执行。使用最先进的AFER工具在帧级别检测单个AU，我们提出了两种新颖的数据结构，用于从单个AU分数中编码AU组合。对应于每种数据结构，采用了两种弱监督学习框架，即多实例学习（MIL）和多聚类实例学习（MCIL），以从视频序列中学习疼痛。实验结果表明，在UNBC - 麦克马斯特肩部疼痛表情数据集上，疼痛识别准确率为87%，曲线下面积（AUC）为0.94。在肺癌患者视频数据集的长视频上进行的测试证明了所提出系统在临床环境中进行疼痛监测的潜在价值。
