Institute of Information Systems, Leuphana University of Lüneburg, Luneburg, Germany
Institute of Information Systems, Leuphana University of Lüneburg, Luneburg, Germany.
Evid Based Ment Health. 2020 Feb;23(1):27-33. doi: 10.1136/ebmental-2019-300135.
Self-reported client assessments during online treatments enable the development of statistical models for the prediction of client improvement and symptom development. Evaluation of these models is mandatory to ensure their validity. For this purpose, we suggest besides a model evaluation based on study data the use of a simulation analysis. The simulation analysis provides insight into the model performance and enables to analyse reasons for a low predictive accuracy. In this study, we evaluate a temporal causal model (TCM) and show that it does not provide reliable predictions of clients' future mood levels. Based on the simulation analysis we investigate the potential reasons for the low predictive performance, for example, noisy measurements and sampling frequency. We conclude that the analysed TCM in its current form is not sufficient to describe the underlying psychological processes. The results demonstrate the importance of model evaluation and the benefit of a simulation analysis. The current manuscript provides practical guidance for conducting model evaluation including simulation analysis.
自我报告的在线治疗中的客户评估使统计模型的开发成为可能，这些模型可以预测客户的改善和症状发展。为了确保模型的有效性，必须对其进行评估。为此，我们建议除了基于研究数据的模型评估外，还使用模拟分析。模拟分析可以深入了解模型的性能，并能够分析预测精度低的原因。在这项研究中，我们评估了一个时间因果模型（TCM），并表明它不能可靠地预测客户未来的情绪水平。基于模拟分析，我们研究了预测性能低的潜在原因，例如噪声测量和采样频率。我们的结论是，分析中的 TCM 目前的形式不足以描述潜在的心理过程。结果表明了模型评估的重要性和模拟分析的好处。本文提供了关于进行模型评估（包括模拟分析）的实用指南。
