Yewale Ashish, Yang Yihui, Nazemifard Neda, Papageorgiou Charles D, Rielly Chris D, Benyahia Brahim
Department of Chemical Engineering, Loughborough University, Loughborough, Leicestershire LE11 3TU, U.K.
Synthetic Molecule Process Development, Process Engineering and Technology, Takeda Pharmaceuticals International Company, 40 Landsdowne Street, Cambridge, Massachusetts 02139, United States.
ACS Eng Au. 2025 May 13;5(3):247-266. doi: 10.1021/acsengineeringau.5c00004. eCollection 2025 Jun 18.
The development of effective synthetic pathways is critical in many industrial sectors. The growing adoption of flow chemistry has opened new opportunities for more cost-effective and environmentally friendly manufacturing technologies. However, the development of effective flow chemistry processes is still hampered by labor- and experiment-intensive methodologies and poor or suboptimal performance. In this context, integrating advanced machine learning strategies into chemical process optimization can significantly reduce experimental burdens and enhance overall efficiency. This paper demonstrates the capabilities of deep reinforcement learning (DRL) as an effective self-optimization strategy for imine synthesis in flow, a key building block in many compounds such as pharmaceuticals and heterocyclic products. A deep deterministic policy gradient (DDPG) agent was designed to iteratively interact with the environment, the flow reactor, and learn how to deliver optimal operating conditions. A mathematical model of the reactor was developed based on new experimental data to train the agent and evaluate alternative self-optimization strategies. To optimize the DDPG agent's training performance, different hyperparameter tuning methods were investigated and compared, including trial-and-error and Bayesian optimization. Most importantly, a novel adaptive dynamic hyperparameter tuning was implemented to further enhance the training performance and optimization outcome of the agent. The performance of the proposed DRL strategy was compared against state-of-the-art gradient-free methods, namely SnobFit and Nelder-Mead. Finally, the outcomes of the different self-optimization strategies were tested experimentally. It was shown that the proposed DDPG agent has superior performance compared to its self-optimization counterparts. It offered better tracking of the global solution and reduced the number of required experiments by approximately 50 and 75% compared to Nelder-Mead and SnobFit, respectively. These findings hold significant promise for the chemical engineering community, offering a robust, efficient, and sustainable approach to optimizing flow chemistry processes and paving the way for broader integration of data-driven methods in process design and operation.
有效的合成途径的开发在许多工业领域都至关重要。流动化学的日益采用为更具成本效益和环境友好的制造技术带来了新机遇。然而，有效的流动化学工艺的开发仍受到劳动和实验密集型方法以及性能不佳或次优的阻碍。在此背景下，将先进的机器学习策略整合到化学过程优化中可显著减轻实验负担并提高整体效率。本文展示了深度强化学习（DRL）作为流动中胺合成的有效自优化策略的能力，胺合成是许多化合物（如药物和杂环产品）中的关键组成部分。设计了一个深度确定性策略梯度（DDPG）智能体，使其与环境（流动反应器）进行迭代交互，并学习如何提供最佳操作条件。基于新的实验数据开发了反应器的数学模型，以训练智能体并评估替代的自优化策略。为了优化DDPG智能体的训练性能，研究并比较了不同的超参数调整方法，包括试错法和贝叶斯优化。最重要的是，实施了一种新颖的自适应动态超参数调整，以进一步提高智能体的训练性能和优化结果。将所提出的DRL策略的性能与最先进的无梯度方法（即SnobFit和Nelder-Mead）进行了比较。最后，对不同自优化策略的结果进行了实验测试。结果表明，所提出的DDPG智能体与其自优化对应物相比具有卓越性能。与Nelder-Mead和SnobFit相比，它能更好地跟踪全局解，并分别将所需实验次数减少了约50%和75%。这些发现为化学工程界带来了重大希望，提供了一种强大、高效且可持续的方法来优化流动化学过程，并为数据驱动方法在过程设计和操作中的更广泛整合铺平了道路。
