Zhang Hongmei, Liang Songshi, Matkovic Luke A, Momin Shadab, Wang Kai, Yang Xiaofeng, Insana Michael F
Key Laboratory of Biomedical Information Engineering of Ministry of Education, School of Life Science and Technology, Xi'an Jiaotong University, Xi'an, China.
Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, China.
Quant Imaging Med Surg. 2023 Aug 1;13(8):4879-4896. doi: 10.21037/qims-22-1147. Epub 2023 Jun 27.
Estimation of the global optima of multiple model parameters is valuable for precisely extracting parameters that characterize a physical environment. This is especially useful for imaging purposes, to form reliable, meaningful physical images with good reproducibility. However, it is challenging to avoid different local minima when the objective function is nonconvex. The problem of global searching of multiple parameters was formulated to be a -D move in the parameter space and the parameter updating scheme was converted to be a state-action decision-making problem.
We proposed a novel Deep Q-learning of Model Parameters (DQMP) method for global optimization which updated the parameter configurations through actions that maximized the Q-value and employed a Deep Reward Network (DRN) designed to learn global reward values from both visible fitting errors and hidden parameter errors. The DRN was constructed with Long Short-Term Memory (LSTM) layers followed by fully connected layers and a rectified linear unit (ReLU) nonlinearity. The depth of the DRN depended on the number of parameters. Through DQMP, the -D parameter search in each step resembled the decision-making of action selections from 3 configurations in a -D board game.
The DQMP method was evaluated by widely used general functions that can express a variety of experimental data and further validated on imaging applications. The convergence of the proposed DRN was evaluated, which showed that the loss values of six general functions all converged after 12 epochs. The parameters estimated by the DQMP method had relative errors of less than 4% for all cases, whereas the relative errors achieved by Q-learning (QL) and the Least Squares Method (LSM) were 17% and 21%, respectively. Furthermore, the imaging experiments demonstrated that the imaging of the parameters estimated by the proposed DQMP method were the closest to the ground truth simulation images when compared to other methods.
The proposed DQMP method was able to achieve global optima, thus yielding accurate model parameter estimates. DQMP is promising for estimating multiple high-dimensional parameters and can be generalized to global optimization for many other complex nonconvex functions and imaging of physical parameters.
估计多个模型参数的全局最优值对于精确提取表征物理环境的参数非常有价值。这对于成像目的尤其有用，以便形成具有良好再现性的可靠、有意义的物理图像。然而，当目标函数非凸时，避免不同的局部最小值具有挑战性。多个参数的全局搜索问题被表述为参数空间中的一维移动，并且参数更新方案被转换为一个状态 - 动作决策问题。
我们提出了一种用于全局优化的新型模型参数深度Q学习（DQMP）方法，该方法通过最大化Q值的动作来更新参数配置，并采用了一个深度奖励网络（DRN），该网络旨在从可见的拟合误差和隐藏的参数误差中学习全局奖励值。DRN由长短期记忆（LSTM）层、全连接层和整流线性单元（ReLU）非线性组成。DRN的深度取决于参数的数量。通过DQMP，每一步中的一维参数搜索类似于在一个三维棋盘游戏中从三种配置中选择动作的决策。
DQMP方法通过广泛使用的能够表达各种实验数据的通用函数进行评估，并在成像应用中进一步验证。对所提出的DRN的收敛性进行了评估，结果表明六个通用函数的损失值在12个epoch后均收敛。DQMP方法估计的参数在所有情况下的相对误差均小于4%，而Q学习（QL）和最小二乘法（LSM）实现的相对误差分别为17%和21%。此外，成像实验表明，与其他方法相比，所提出的DQMP方法估计的参数成像最接近真实模拟图像。
所提出的DQMP方法能够实现全局最优，从而产生准确的模型参数估计。DQMP在估计多个高维参数方面很有前景，并且可以推广到许多其他复杂非凸函数的全局优化以及物理参数成像。
