Department of Computer Science, University of Kentucky, 329 Rose Street, Lexington, KY 40506, USA.
Department of Computer Science, University of Kentucky, 329 Rose Street, Lexington, KY 40506, USA; Division of Biomedical Informatics, Department of Internal Medicine, University Kentucky, 725 Rose Street, Lexington, KY 40536, USA.
J Biomed Inform. 2017 Nov;75S:S85-S93. doi: 10.1016/j.jbi.2017.05.008. Epub 2017 May 12.
The CEGS N-GRID 2016 Shared Task in Clinical Natural Language Processing (NLP) provided a set of 1000 neuropsychiatric notes to participants as part of a competition to predict psychiatric symptom severity scores. This paper summarizes our methods, results, and experiences based on our participation in the second track of the shared task.
Classical methods of text classification usually fall into one of three problem types: binary, multi-class, and multi-label classification. In this effort, we study ordinal regression problems with text data where misclassifications are penalized differently based on how far apart the ground truth and model predictions are on the ordinal scale. Specifically, we present our entries (methods and results) in the N-GRID shared task in predicting research domain criteria (RDoC) positive valence ordinal symptom severity scores (absent, mild, moderate, and severe) from psychiatric notes.
We propose a novel convolutional neural network (CNN) model designed to handle ordinal regression tasks on psychiatric notes. Broadly speaking, our model combines an ordinal loss function, a CNN, and conventional feature engineering (wide features) into a single model which is learned end-to-end. Given interpretability is an important concern with nonlinear models, we apply a recent approach called locally interpretable model-agnostic explanation (LIME) to identify important words that lead to instance specific predictions.
Our best model entered into the shared task placed third among 24 teams and scored a macro mean absolute error (MMAE) based normalized score (100·(1-MMAE)) of 83.86. Since the competition, we improved our score (using basic ensembling) to 85.55, comparable with the winning shared task entry. Applying LIME to model predictions, we demonstrate the feasibility of instance specific prediction interpretation by identifying words that led to a particular decision.
In this paper, we present a method that successfully uses wide features and an ordinal loss function applied to convolutional neural networks for ordinal text classification specifically in predicting psychiatric symptom severity scores. Our approach leads to excellent performance on the N-GRID shared task and is also amenable to interpretability using existing model-agnostic approaches.
CEGS N-GRID 2016 临床自然语言处理（NLP）共享任务为参与者提供了一组 1000 份神经精神病学笔记，作为预测精神症状严重程度评分竞赛的一部分。本文总结了我们基于参与共享任务第二轨道的方法、结果和经验。
经典的文本分类方法通常属于以下三种问题类型之一：二分类、多分类和多标签分类。在这项工作中，我们研究了文本数据中的有序回归问题，其中基于真实值和模型预测值在有序尺度上的差异，对错误分类进行不同的惩罚。具体来说，我们展示了我们在 N-GRID 共享任务中的参赛作品，用于从精神病学笔记中预测研究领域标准（RDoC）正价序症状严重程度评分（不存在、轻度、中度和重度）。
我们提出了一种新颖的卷积神经网络（CNN）模型，旨在处理精神病学笔记上的有序回归任务。广义而言，我们的模型结合了有序损失函数、CNN 和传统的特征工程（宽特征），形成一个端到端学习的单一模型。由于非线性模型的可解释性是一个重要的关注点，我们应用了一种最近的方法，称为局部可解释模型不可知解释（LIME），来识别导致特定实例预测的重要单词。
我们提交的最佳模型在 24 个参赛团队中排名第三，在共享任务中获得了 83.86 的宏平均绝对误差（MMAE）归一化得分（100·（1-MMAE））。自竞赛以来，我们通过使用基本集成的方法将分数提高到了 85.55，与共享任务的获胜参赛作品相当。通过对模型预测应用 LIME，我们通过识别导致特定决策的单词，展示了实例特定预测解释的可行性。
在本文中，我们提出了一种成功地使用宽特征和有序损失函数应用于卷积神经网络的方法，用于特定的有序文本分类，特别是预测精神症状严重程度评分。我们的方法在 N-GRID 共享任务中表现出色，并且也可以使用现有的模型不可知方法进行可解释性。
