使用胶囊网络增强手术室无接触界面的手术手势识别。

Enhancement of surgical hand gesture recognition using a capsule network for a contactless interface in the operating room.

机构信息

Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea.

Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea; Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea.

出版信息

Comput Methods Programs Biomed. 2020 Jul;190:105385. doi: 10.1016/j.cmpb.2020.105385. Epub 2020 Feb 6.

DOI:10.1016/j.cmpb.2020.105385

PMID:32062090

Abstract

BACKGROUND AND OBJECTIVE

Hand gesture recognition systems in operating rooms (ORs) are crucial for browsing and controlling computer-aided devices, which have been developed to decrease the risk of contamination during surgical procedures.

METHODS

We proposed the use of hand gesture recognition to enhance accuracies and recognition areas with the capsule network (CapsNet) of deep neural network and Leap Motionâ Our method includes the i) extraction and preprocessing of infrared (IR) images (60 frames per second) from Leap Motion™, ii) training of various types of networks, and iii) gesture recognition evaluation in the OR. We trained the images of training dataset (N=903) and tested images (N=100) using five types of surgical hand gestures including hovering, grab, click, one peak, and two peaks by 10 subjects with various types of augmentation methods including rotate (0, 90, 180), scale, translation, illumination, and resize.

RESULTS

CapsNet achieved a classification accuracy of 86.46% (around 10% improvement) compared with 73.67% for the baseline convolutional neural network (CNN) and 76.4% for VGG16.

CONCLUSIONS

In conclusion, the accuracy of hand gesture recognition with CapsNet was better than that of conventional CNNs, which could be used to navigate and manipulate various types of computer-aided devices and applications through contactless gesture interaction.

摘要

背景与目的

手术室中的手势识别系统对于浏览和控制计算机辅助设备至关重要，这些设备的开发旨在降低手术过程中的污染风险。

方法

我们提出使用手势识别技术，通过深度神经网络的胶囊网络（CapsNet）和 Leap Motion 来提高准确性和识别区域。我们的方法包括：i）从 Leap Motion™ 提取和预处理红外（IR）图像（每秒 60 帧），ii）训练各种类型的网络，以及 iii）在手术室中进行手势识别评估。我们使用五种类型的手术手势（包括悬停、抓取、点击、单峰和双峰）对 903 张训练数据集图像和 100 张测试图像进行了训练，其中包括旋转（0、90、180）、缩放、平移、光照和调整大小等多种增强方法，由 10 位具有不同类型的受试者进行。