Embedded Vision Systems Group, Computer Vision Laboratory, Department of Automatic Control and Robotics, AGH University of Science and Technology, Al. Mickiewicza 30, 30-059 Krakow, Poland.
Sensors (Basel). 2022 Jul 3;22(13):5017. doi: 10.3390/s22135017.
The information about optical flow, i.e., the movement of pixels between two consecutive images from a video sequence, is used in many vision systems, both classical and those based on deep neural networks. In some robotic applications, e.g., in autonomous vehicles, it is necessary to calculate the flow in real time. This represents a challenging task, especially for high-resolution video streams. In this work, two gradient-based algorithms-Lucas-Kanade and Horn-Schunck-were implemented on a ZCU 104 platform with Xilinx Zynq UltraScale+ MPSoC FPGA. A vector data format was used to enable flow calculation for a 4K (Ultra HD, 3840 × 2160 pixels) video stream at 60 fps. In order to detect larger pixel displacements, a multi-scale approach was used in both algorithms. Depending on the scale, the calculations were performed for different data formats, allowing for more efficient processing by reducing resource utilisation. The presented solution allows real-time optical flow determination in multiple scales for a 4K resolution with estimated energy consumption below 6 W. The algorithms realised in this work can be a component of a larger vision system in advanced surveillance systems or autonomous vehicles.
光流信息，即视频序列中两帧连续图像之间的像素运动，被广泛应用于传统视觉系统和基于深度神经网络的视觉系统中。在一些机器人应用中，例如在自动驾驶汽车中，需要实时计算光流。这是一项具有挑战性的任务，尤其是对于高分辨率视频流而言。在这项工作中，基于梯度的 Lucas-Kanade 和 Horn-Schunck 两种算法在 ZCU 104 平台上的 Xilinx Zynq UltraScale+MPSoC FPGA 上实现。使用向量数据格式可以实现每秒 60 帧的 4K（超高清，3840×2160 像素）视频流的光流计算。为了检测更大的像素位移，两种算法都使用了多尺度方法。根据尺度，针对不同的数据格式进行计算，通过减少资源利用，实现更高效的处理。所提出的解决方案允许以 4K 分辨率进行多尺度实时光流确定，估计能耗低于 6W。在这项工作中实现的算法可以成为高级监控系统或自动驾驶汽车中更大视觉系统的一个组成部分。
