Wu Jianhua, Gao Bingzhao, Gao Jincheng, Yu Jianhao, Chu Hongqing, Yu Qiankun, Gong Xun, Chang Yi, Tseng H Eric, Chen Hong, Chen Jie
School of Automotive Studies, Tongji University, Shanghai 201804, China.
Frontiers Science Center for Intelligent Autonomous Systems, Tongji University, Shanghai 201210, China.
Research (Wash D C). 2024 Jul 16;7:0399. doi: 10.34133/research.0399. eCollection 2024.
With the development of artificial intelligence and breakthroughs in deep learning, large-scale foundation models (FMs), such as generative pre-trained transformer (GPT), Sora, etc., have achieved remarkable results in many fields including natural language processing and computer vision. The application of FMs in autonomous driving holds considerable promise. For example, they can contribute to enhancing scene understanding and reasoning. By pre-training on rich linguistic and visual data, FMs can understand and interpret various elements in a driving scene, and provide cognitive reasoning to give linguistic and action instructions for driving decisions and planning. Furthermore, FMs can augment data based on the understanding of driving scenarios to provide feasible scenes of those rare occurrences in the long tail distribution that are unlikely to be encountered during routine driving and data collection. The enhancement can subsequently lead to improvement in the accuracy and reliability of autonomous driving systems. Another testament to the potential of FMs' applications lies in world models, exemplified by the DREAMER series, which showcases the ability to comprehend physical laws and dynamics. Learning from massive data under the paradigm of self-supervised learning, world models can generate unseen yet plausible driving environments, facilitating the enhancement in the prediction of road users' behaviors and the off-line training of driving strategies. In this paper, we synthesize the applications and future trends of FMs in autonomous driving. By utilizing the powerful capabilities of FMs, we strive to tackle the potential issues stemming from the long-tail distribution in autonomous driving, consequently advancing overall safety in this domain.
随着人工智能的发展以及深度学习的突破，大规模基础模型（FMs），如生成式预训练变换器（GPT）、Sora等，在自然语言处理和计算机视觉等许多领域都取得了显著成果。FMs在自动驾驶中的应用前景广阔。例如，它们有助于增强场景理解和推理能力。通过在丰富的语言和视觉数据上进行预训练，FMs能够理解和解释驾驶场景中的各种元素，并提供认知推理，为驾驶决策和规划给出语言和行动指令。此外，FMs可以基于对驾驶场景的理解来扩充数据，以提供常规驾驶和数据收集过程中不太可能遇到的长尾分布中那些罕见情况的可行场景。这种增强随后可以提高自动驾驶系统的准确性和可靠性。FMs应用潜力的另一个证明在于世界模型，以DREAMER系列为例，它展示了理解物理定律和动力学的能力。在自监督学习范式下从海量数据中学习，世界模型可以生成未见但合理的驾驶环境，有助于增强对道路使用者行为的预测以及驾驶策略的离线训练。在本文中，我们综合了FMs在自动驾驶中的应用和未来趋势。通过利用FMs的强大功能，我们努力解决自动驾驶中长尾分布带来的潜在问题，从而提高该领域的整体安全性。
