Beijing Key Laboratory for Green Catalysis and Separation and Department of Chemical Engineering, College of Materials Science & Engineering, Beijing University of Technology, Beijing 100124, P. R. China.
Engineering Research Center of Digital Community, Ministry of Education, Beijing Laboratory for Urban Mass Transit and Beijing Key Laboratory of Computational Intelligence and Intelligence System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, P. R. China.
J Chem Inf Model. 2024 Jul 8;64(13):4958-4965. doi: 10.1021/acs.jcim.4c00065. Epub 2024 Mar 26.
Along with the development of machine learning, deep learning, and large language models (LLMs) such as GPT-4 (GPT: Generative Pre-Trained Transformer), artificial intelligence (AI) tools have been playing an increasingly important role in chemical and material research to facilitate the material screening and design. Despite the exciting progress of GPT-4 based AI research assistance, open-source LLMs have not gained much attention from the scientific community. This work primarily focused on metal-organic frameworks (MOFs) as a subdomain of chemistry and evaluated six top-rated open-source LLMs with a comprehensive set of tasks including MOFs knowledge, basic chemistry knowledge, in-depth chemistry knowledge, knowledge extraction, database reading, predicting material property, experiment design, computational scripts generation, guiding experiment, data analysis, and paper polishing, which covers the basic units of MOFs research. In general, these LLMs were capable of most of the tasks. Especially, Llama2-7B and ChatGLM2-6B were found to perform particularly well with moderate computational resources. Additionally, the performance of different parameter versions of the same model was compared, which revealed the superior performance of higher parameter versions.
随着机器学习、深度学习和大型语言模型（如 GPT-4，GPT：生成式预训练转换器）等技术的发展，人工智能（AI）工具在化学和材料研究中发挥着越来越重要的作用，有助于进行材料筛选和设计。尽管基于 GPT-4 的人工智能研究辅助工具取得了令人兴奋的进展，但开源大型语言模型并未引起科学界的太多关注。这项工作主要集中在金属有机骨架（MOFs）作为化学的一个分支领域，并评估了六个顶级的开源大型语言模型，这些模型涵盖了 MOFs 研究的基本单元，通过一套全面的任务进行评估，包括 MOFs 知识、基础化学知识、深入化学知识、知识提取、数据库阅读、预测材料性质、实验设计、计算脚本生成、实验指导、数据分析和论文润色。总的来说，这些大型语言模型能够胜任大部分任务。特别是，Llama2-7B 和 ChatGLM2-6B 被发现能够在适度的计算资源下表现得尤为出色。此外，还比较了同一模型不同参数版本的性能，结果表明更高参数版本的性能更优。
