Groth Colin, Fricke Sascha, Castillo Susana, Magnor Marcus
IEEE Trans Vis Comput Graph. 2023 May;29(5):2508-2516. doi: 10.1109/TVCG.2023.3247080. Epub 2023 Mar 29.
In this paper, we propose a wavelet-based video codec specifically designed for VR displays that enables real-time playback of high-resolution 360° videos. Our codec exploits the fact that only a fraction of the full 360° video frame is visible on the display at any time. To load and decode the video viewport-dependently in real time, we make use of the wavelet transform for intra- as well as inter-frame coding. Thereby, the relevant content is directly streamed from the drive, without the need to hold the entire frames in memory. With an average of 193 frames per second at 8192 × 8192 -pixel full-frame resolution, the conducted evaluation demonstrates that our codec's decoding performance is up to 272% higher than that of the state-of-the-art video codecs H.265 and AV1 for typical VR displays. By means of a perceptual study, we further illustrate the necessity of high frame rates for a better VR experience. Finally, we demonstrate how our wavelet-based codec can also directly be used in conjunction with foveation for further performance increase.
在本文中，我们提出了一种专门为虚拟现实（VR）显示器设计的基于小波的视频编解码器，它能够实时播放高分辨率360°视频。我们的编解码器利用了这样一个事实：在任何时刻，显示器上仅可见完整360°视频帧的一小部分。为了实时地根据视频视口进行加载和解码，我们在帧内和帧间编码中都使用了小波变换。由此，相关内容直接从驱动器进行流式传输，无需将整个帧保存在内存中。在8192×8192像素全帧分辨率下平均每秒193帧，所进行的评估表明，对于典型的VR显示器，我们编解码器的解码性能比最先进的视频编解码器H.265和AV1高出272%。通过一项感知研究，我们进一步说明了高帧率对于获得更好VR体验的必要性。最后，我们展示了基于小波的编解码器如何也能直接与注视点技术结合使用以进一步提高性能。
