National Centre for Computational Design and Discovery of Novel Materials (MARVEL), École Polytechnique Fédérale de Lausanne, CH-1015, Lausanne, Switzerland.
Theory and Simulation of Materials (THEOS), Faculté des Sciences et Techniques de l'Ingénieur, École Polytechnique Fédérale de Lausanne, CH-1015, Lausanne, Switzerland.
Sci Data. 2020 Sep 8;7(1):300. doi: 10.1038/s41597-020-00638-4.
The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA's workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with external simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.
计算能力的不断提高和先进计算方法的持续发展为最近的科学进步做出了很大贡献。这些发展带来了新的挑战，需要处理大量的计算和数据。下一代百亿亿次级超级计算机将加剧这些挑战，因此自动化和可扩展的解决方案变得至关重要。近年来，我们一直在开发 AiiDA（aiida.net），这是一个强大的开源高通量基础设施，旨在解决自动化工作流程管理和数据来源记录需求所带来的挑战。在这里，我们介绍了实现持续性能所需的开发和功能，AiiDA 支持每小时处理数万个流程，同时自动保存和存储完整的数据来源，使其可查询和遍历，从而实现高性能数据分析。AiiDA 的工作流程语言提供了高级自动化、错误处理功能和灵活的插件模型，允许与外部模拟软件接口。相关的插件注册使扩展的无缝共享成为可能，为一个致力于使模拟更强大、更用户友好和可重复的充满活力的用户社区提供支持。
