School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
Room 805, School of Mechanical Engineering, Shanghai Jiao Tong University, Dongchuan Road 800, Minhang District, Shanghai, 200240, China.
Int J Comput Assist Radiol Surg. 2019 Dec;14(12):2147-2154. doi: 10.1007/s11548-019-02031-y. Epub 2019 Jul 12.
Image-guided surgical navigation system (SNS) has proved to be an increasingly important assistance tool for mini-invasive surgery. However, using standard devices such as keyboard and mouse as human-computer interaction (HCI) is a latent vector of infectious medium, causing risks to patients and surgeons. To solve the human-computer interaction problem, we proposed an optimized structure of LSTM based on a depth camera to recognize gestures and applied it to an in-house oral and maxillofacial surgical navigation system (Qin et al. in Int J Comput Assist Radiol Surg 14(2):281-289, 2019).
METHODS: The proposed optimized structure of LSTM named multi-LSTM allows multiple input layers and takes into account the relationships between inputs. To combine the gesture recognition with the SNS, four left-hand signs waving along four directions were designed to correspond to four operations of the mouse, and the motion of right hand was used to control the movement of the cursor. Finally, a phantom study for zygomatic implant placement was conducted to evaluate the feasibility of multi-LSTM as HCI. RESULTS: 3D hand trajectories of both wrist and elbow from 10 participants were collected to train the recognition network. Then tenfold cross-validation was performed for judging signs, and the mean accuracy was 96% ± 3%. In the phantom study, four implants were successfully placed, and the average deviations of planned-placed implants were 1.22 mm and 1.70 mm for the entry and end points, respectively, while the angular deviation ranged from 0.4° to 2.9°.
The results showed that this non-contact user interface based on multi-LSTM could be used as a promising tool to eliminate the disinfection problem in operation room and alleviate manipulation complexity of surgical navigation system.
图像引导手术导航系统（SNS）已被证明是微创手术中越来越重要的辅助工具。然而，使用键盘和鼠标等标准设备作为人机交互（HCI）存在感染媒介的潜在风险，会给患者和外科医生带来风险。为了解决人机交互问题，我们提出了一种基于深度相机的优化 LSTM 结构来识别手势，并将其应用于内部口腔颌面外科导航系统（Qin 等人，Int J Comput Assist Radiol Surg 14（2）：281-289，2019）。
所提出的优化 LSTM 结构命名为多 LSTM，允许多个输入层，并考虑输入之间的关系。为了将手势识别与 SNS 结合，设计了四个沿四个方向挥手的左手标志，分别对应鼠标的四个操作，而右手的运动用于控制光标移动。最后，进行了颧骨种植体放置的体模研究，以评估多 LSTM 作为 HCI 的可行性。
从 10 名参与者收集了手腕和肘部的 3D 手部轨迹，以训练识别网络。然后进行了十折交叉验证来判断标志，平均准确率为 96%±3%。在体模研究中，成功放置了四个植入物，计划放置植入物的平均偏差为入口和终点处的 1.22mm 和 1.70mm，而角度偏差范围为 0.4°至 2.9°。
结果表明，这种基于多 LSTM 的非接触式用户界面可以作为一种有前途的工具，用于消除手术室的消毒问题，并减轻手术导航系统的操作复杂性。
