Cosco Francesco, Garre Carlos, Bruno Fabio, Muzzupappa Maurizio, Otaduy Miguel A
IEEE Trans Vis Comput Graph. 2013 Jan;19(1):159-72. doi: 10.1109/TVCG.2012.107. Epub 2012 Apr 17.
Visuo-haptic mixed reality consists of adding to a real scene the ability to see and touch virtual objects. It requires the use of see-through display technology for visually mixing real and virtual objects, and haptic devices for adding haptic interaction with the virtual objects. Unfortunately, the use of commodity haptic devices poses obstruction and misalignment issues that complicate the correct integration of a virtual tool and the user's real hand in the mixed reality scene. In this work, we propose a novel mixed reality paradigm where it is possible to touch and see virtual objects in combination with a real scene, using commodity haptic devices, and with a visually consistent integration of the user's hand and the virtual tool. We discuss the visual obstruction and misalignment issues introduced by commodity haptic devices, and then propose a solution that relies on four simple technical steps: color-based segmentation of the hand, tracking-based segmentation of the haptic device, background repainting using image-based models, and misalignment-free compositing of the user's hand. We have developed a successful proof-of-concept implementation, where a user can touch virtual objects and interact with them in the context of a real scene, and we have evaluated the impact on user performance of obstruction and misalignment correction.
视觉触觉混合现实包括在真实场景中增加看到和触摸虚拟物体的能力。它需要使用透视显示技术来在视觉上混合真实和虚拟物体，以及使用触觉设备来增加与虚拟物体的触觉交互。不幸的是，使用商用触觉设备会带来遮挡和对齐问题，这使得在混合现实场景中虚拟工具与用户真实手部的正确集成变得复杂。在这项工作中，我们提出了一种新颖的混合现实范式，在该范式中，使用商用触觉设备，并在用户手部与虚拟工具的视觉上一致的集成情况下，能够结合真实场景触摸和看到虚拟物体。我们讨论了商用触觉设备引入的视觉遮挡和对齐问题，然后提出了一种基于四个简单技术步骤的解决方案：基于颜色的手部分割、基于跟踪的触觉设备分割、使用基于图像的模型进行背景重绘以及用户手部的无对齐合成。我们已经开发出一个成功的概念验证实现，在其中用户可以在真实场景中触摸虚拟物体并与之交互，并且我们已经评估了遮挡和对齐校正对用户性能的影响。
