University of Leuven (KU Leuven), Department for Pharmaceutical and Pharmacological Sciences, Pharmaceutical Analysis, Herestraat 49, 3000 Leuven, Belgium.
Vrije Universiteit Brussel, Department of Computer Science, Artificial Intelligence Lab, Pleinlaan 9, 1050 Brussel, Belgium.
J Chromatogr A. 2021 Feb 8;1638:461900. doi: 10.1016/j.chroma.2021.461900. Epub 2021 Jan 13.
An important challenge in chromatography is the development of adequate separation methods. Accurate retention models can significantly simplify and expedite the development of adequate separation methods for complex mixtures. The purpose of this study was to introduce reinforcement learning to chromatographic method development, by training a double deep Q-learning algorithm to select optimal isocratic scouting runs to generate accurate retention models. These scouting runs were fit to the Neue-Kuss retention model, which was then used to predict retention factors both under isocratic and gradient conditions. The quality of these predictions was compared to experimental data points, by computing a mean relative percentage error (MRPE) between the predicted and actual retention factors. By providing the reinforcement learning algorithm with a reward whenever the scouting runs led to accurate retention models and a penalty when the analysis time of a selected scouting run was too high (> 1h); it was hypothesized that the reinforcement learning algorithm should by time learn to select good scouting runs for compounds displaying a variety of characteristics. The reinforcement learning algorithm developed in this work was first trained on simulated data, and then evaluated on experimental data for 57 small molecules - each run at 10 different fractions of organic modifier (0.05 to 0.90) and four different linear gradients. The results showed that the MRPE of these retention models (3.77% for isocratic runs and 1.93% for gradient runs), mostly obtained via 3 isocratic scouting runs for each compound, were comparable in performance to retention models obtained by fitting the Neue-Kuss model to all (10) available isocratic datapoints (3.26% for isocratic runs and 4.97% for gradient runs) and retention models obtained via a "chromatographer's selection" of three scouting runs (3.86% for isocratic runs and 6.66% for gradient runs). It was therefore concluded that the reinforcement learning algorithm learned to select optimal scouting runs for retention modeling, by selecting 3 (out of 10) isocratic scouting runs per compound, that were informative enough to successfully capture the retention behavior of each compound.
色谱学中的一个重要挑战是开发合适的分离方法。准确的保留模型可以大大简化和加快复杂混合物的合适分离方法的开发。本研究的目的是将强化学习引入色谱方法开发中，通过训练双深度 Q 学习算法来选择最佳等度预实验运行，以生成准确的保留模型。这些预实验运行拟合到 Neue-Kuss 保留模型中，然后用于预测等度和梯度条件下的保留因子。通过计算预测和实际保留因子之间的平均相对百分比误差 (MRPE)，将这些预测的质量与实验数据点进行比较。通过在每次预实验运行导致准确的保留模型时为强化学习算法提供奖励，并在选择的预实验运行的分析时间过高 (>1 小时) 时给予惩罚；假设强化学习算法应该随着时间的推移学会为显示各种特征的化合物选择好的预实验运行。本工作中开发的强化学习算法首先在模拟数据上进行训练，然后在 57 个小分子的实验数据上进行评估-每个运行在 10 个不同的有机改性剂分数(0.05 至 0.90)和四个不同的线性梯度。结果表明，这些保留模型的 MRPE(等度运行时为 3.77%，梯度运行时为 1.93%)，主要通过为每个化合物进行 3 次等度预实验运行获得，与通过拟合 Neue-Kuss 模型获得的所有(10)个可用等度数据点的保留模型(等度运行时为 3.26%，梯度运行时为 4.97%)和通过"色谱师选择"获得的保留模型(3.86%等度运行和 6.66%梯度运行)的性能相当。因此，得出结论，强化学习算法通过为每个化合物选择 3(10 个中的 3 个)等度预实验运行，学会选择用于保留建模的最佳预实验运行，这些预实验运行足以成功捕获每个化合物的保留行为。
