Nguyen Duy-Anh, Li Minyi, Lambert Gavin, Kowalczyk Ryszard, McDonald Rachael, Vo Quoc Bao
School of Software and Electrical Engineering, Swinburne University of Technology, Hawthorn, Australia.
School of Computing Technologies, RMIT, Melbourne, Australia.
JMIR Form Res. 2024 Mar 25;8:e52482. doi: 10.2196/52482.
Extractive methods for machine reading comprehension (MRC) tasks have achieved comparable or better accuracy than human performance on benchmark data sets. However, such models are not as successful when adapted to complex domains such as health care. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared with an average open-domain context. This causes the MRC model to make less accurate and slower predictions. A potential solution to this problem is to reduce the input context of the MRC model by extracting only the necessary parts from the original context.
This study aims to develop a method for extracting useful contexts from long articles as an additional component to the question answering task, enabling the MRC model to work more efficiently and accurately.
Existing approaches to context extraction in MRC are based on sentence selection strategies, in which the models are trained to find the sentences containing the answer. We found that using only the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relationship between the usefulness of the context and the confidence score output of the MRC model. Our investigation showed that a precise input context can boost the prediction correctness of the MRC and greatly reduce inference time. We proposed a method to estimate the utility of each sentence in a context in answering the question and then extract a new, shorter context according to these estimations. We generated a data set to train 2 models for estimating sentence utility, based on which we selected more precise contexts that improved the MRC model's performance.
We demonstrated our approach on the Question Answering Data Set for COVID-19 and Biomedical Semantic Indexing and Question Answering data sets and showed that the approach benefits the downstream MRC model. First, the method substantially reduced the inference time of the entire question answering system by 6 to 7 times. Second, our approach helped the MRC model predict the answer more correctly compared with using the original context (F-score increased from 0.724 to 0.744 for the Question Answering Data Set for COVID-19 and from 0.651 to 0.704 for the Biomedical Semantic Indexing and Question Answering). We also found a potential problem where extractive transformer MRC models predict poorly despite being given a more precise context in some cases.
The proposed context extraction method allows the MRC model to achieve improved prediction correctness and a significantly reduced MRC inference time. This approach works technically with any MRC model and has potential in tasks involving processing long texts.
用于机器阅读理解（MRC）任务的提取方法在基准数据集上已取得与人类表现相当或更好的准确率。然而，当应用于医疗保健等复杂领域时，此类模型并不那么成功。主要原因之一是，与一般开放域上下文相比，MRC模型在复杂领域运行时需要处理的上下文可能要大得多。这导致MRC模型做出的预测准确性降低且速度变慢。解决此问题的一个潜在方法是通过仅从原始上下文中提取必要部分来减少MRC模型的输入上下文。
本研究旨在开发一种从长篇文章中提取有用上下文的方法，作为问答任务的一个附加组件，使MRC模型能够更高效、准确地工作。
MRC中现有的上下文提取方法基于句子选择策略，即训练模型找到包含答案的句子。我们发现仅使用包含答案的句子不足以让MRC模型正确预测。我们进行了一系列实证研究，并观察到上下文的有用性与MRC模型输出的置信度得分之间存在密切关系。我们的调查表明，精确的输入上下文可以提高MRC的预测正确性并大大减少推理时间。我们提出了一种方法来估计上下文中每个句子在回答问题时的效用，然后根据这些估计提取一个新的、更短的上下文。我们生成了一个数据集来训练2个用于估计句子效用的模型，并在此基础上选择了更精确的上下文，从而提高了MRC模型的性能。
我们在新冠肺炎问答数据集以及生物医学语义索引与问答数据集上展示了我们的方法，并表明该方法对下游MRC模型有益。首先，该方法将整个问答系统的推理时间大幅减少了6至7倍。其次，与使用原始上下文相比，我们的方法帮助MRC模型更正确地预测答案（新冠肺炎问答数据集的F值从0.724提高到0.744，生物医学语义索引与问答数据集的F值从0.651提高到0.704）。我们还发现了一个潜在问题，即在某些情况下，尽管给定了更精确的上下文，但抽取式变压器MRC模型的预测效果仍然不佳。
所提出的上下文提取方法使MRC模型能够提高预测正确性，并显著减少MRC推理时间。该方法在技术上适用于任何MRC模型，并且在涉及处理长文本的任务中具有潜力。
