Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China.
College of Zhicheng, Fuzhou University, Fuzhou 350108, China.
Sensors (Basel). 2022 Jul 23;22(15):5502. doi: 10.3390/s22155502.
The booming haptic data significantly improve the users' immersion during multimedia interaction. As a result, the study of a Haptic-based Interaction System has attracted the attention of the multimedia community. To construct such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to the user experience. Despite audio-visual synchronization efforts, there is still a lack of a haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic-visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic-visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics and a key frame extraction of visual signals based on deep-object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic-visual signals. Without timestamps in the signal flow, the proposed method is still effective and more robust in complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.
蓬勃发展的触觉数据显著提高了多媒体交互过程中的用户沉浸感。因此，基于触觉的交互系统的研究引起了多媒体界的关注。要构建这样的系统，一个具有挑战性的任务是同步多个感觉信号，这对用户体验至关重要。尽管已经进行了音频-视觉同步的努力，但仍然缺乏触觉感知的多媒体同步模型。在这项工作中，我们提出了一种与时间戳无关的触觉-视觉信号传输同步方法。首先，我们利用触觉-视觉通信系统传输和回放过程中的顺序相关性。其次，我们基于力反馈特性开发了触觉信号的关键样本提取方法，并基于深度目标检测开发了视觉信号的关键帧提取方法。最后，我们将关键样本和关键帧结合起来，以实现相应的触觉-视觉信号同步。在信号流中没有时间戳的情况下，该方法在复杂的网络条件下仍然有效且更具鲁棒性。主观评估还表明，该方法显著提高了用户体验。
