McKinney-Bock Katy, Bedrick Steven
Center for Spoken Language Understanding, Oregon Health & Science University, Portland, Oregon, USA.
Proc Conf. 2019 Jun;2019(RepEval):52-62. doi: 10.18653/v1/w19-2007.
In clinical assessment of people with aphasia, impairment in the ability to recall and produce words for objects () is assessed using a confrontation naming task, where a target stimulus is viewed and a corresponding label is spoken by the participant. Vector space word embedding models have had inital results in assessing semantic similarity of target-production pairs in order to automate scoring of this task; however, the resulting models are also highly dependent upon training parameters. To select an optimal family of models, we fit a beta regression model to the distribution of performance metrics on a set of 2,880 grid search models and evaluate the resultant first- and second-order effects to explore how parameterization affects model performance. Comparing to SimLex-999, we show that clinical data can be used in an evaluation task with comparable optimal parameter settings as standard NLP evaluation datasets.
在对失语症患者进行临床评估时，使用对答命名任务评估其回忆和说出物体对应词汇的能力受损情况，即让参与者观看目标刺激物并说出相应的标签。向量空间词嵌入模型在评估目标 - 产出对的语义相似性以实现该任务的自动评分方面取得了初步成果；然而，所得模型也高度依赖于训练参数。为了选择最优的模型族，我们将一个贝塔回归模型拟合到一组2880个网格搜索模型的性能指标分布上，并评估所得的一阶和二阶效应，以探究参数化如何影响模型性能。与SimLex - 999相比，我们表明临床数据可用于具有与标准自然语言处理评估数据集相当的最优参数设置的评估任务中。
