Swift Melissa E, Ayers Wyatt, Pallanck Sophie, Wehrwein Scott
IEEE Trans Vis Comput Graph. 2023 Jan;29(1):171-181. doi: 10.1109/TVCG.2022.3209454. Epub 2022 Dec 20.
What can we learn about a scene by watching it for months or years? A video recorded over a long timespan will depict interesting phenomena at multiple timescales, but identifying and viewing them presents a challenge. The video is too long to watch in full, and some things are too slow to experience in real-time, such as glacial retreat or the gradual shift from summer to fall. Timelapse videography is a common approach to summarizing long videos and visualizing slow timescales. However, a timelapse is limited to a single chosen temporal frequency, and often appears flickery due to aliasing. Also, the length of the timelapse video is directly tied to its temporal resolution, which necessitates tradeoffs between those two facets. In this paper, we propose Video Temporal Pyramids, a technique that addresses these limitations and expands the possibilities for visualizing the passage of time. Inspired by spatial image pyramids from computer vision, we developed an algorithm that builds video pyramids in the temporal domain. Each level of a Video Temporal Pyramid visualizes a different timescale; for instance, videos from the monthly timescale are usually good for visualizing seasonal changes, while videos from the one-minute timescale are best for visualizing sunrise or the movement of clouds across the sky. To help explore the different pyramid levels, we also propose a Video Spectrogram to visualize the amount of activity across the entire pyramid, providing a holistic overview of the scene dynamics and the ability to explore and discover phenomena across time and timescales. To demonstrate our approach, we have built Video Temporal Pyramids from ten outdoor scenes, each containing months or years of data. We compare Video Temporal Pyramid layers to naive timelapse and find that our pyramids enable alias-free viewing of longer-term changes. We also demonstrate that the Video Spectrogram facilitates exploration and discovery of phenomena across pyramid levels, by enabling both overview and detail-focused perspectives.
通过数月或数年观察一个场景，我们能了解到什么？长时间跨度录制的视频会在多个时间尺度上呈现有趣的现象，但识别和观看这些现象具有挑战性。视频太长无法完整观看，有些事情实时体验又太慢，比如冰川消退或从夏季到秋季的逐渐转变。延时摄影是总结长视频和可视化慢速时间尺度的常用方法。然而，延时摄影限于单一选定的时间频率，并且由于混叠常常显得闪烁。此外，延时视频的长度直接与其时间分辨率相关，这就需要在这两个方面进行权衡。在本文中，我们提出了视频时间金字塔，一种解决这些限制并扩展时间流逝可视化可能性的技术。受计算机视觉中的空间图像金字塔启发，我们开发了一种在时间域构建视频金字塔的算法。视频时间金字塔的每一层都可视化不同的时间尺度；例如，月度时间尺度的视频通常有利于可视化季节变化，而一分钟时间尺度的视频最适合可视化日出或云朵在天空中的移动。为了帮助探索不同的金字塔层级，我们还提出了视频频谱图来可视化整个金字塔的活动量，提供场景动态的整体概述以及跨时间和时间尺度探索和发现现象的能力。为了展示我们的方法，我们从十个户外场景构建了视频时间金字塔，每个场景包含数月或数年的数据。我们将视频时间金字塔层与简单的延时摄影进行比较，发现我们的金字塔能够实现无混叠地观看长期变化。我们还证明，视频频谱图通过提供概览和聚焦细节的视角，促进了跨金字塔层级对现象的探索和发现。
