School of Informatics, Xiamen University, China.
School of Informatics, Xiamen University, China.
Neural Netw. 2024 May;173:106217. doi: 10.1016/j.neunet.2024.106217. Epub 2024 Feb 27.
Recently, cross-lingual transfer learning has attracted extensive attention from both academia and industry. Previous studies usually focus only on the single-level alignment (e.g., word-level, sentence-level), based on pre-trained language models. However, it leads to suboptimal performance in downstream tasks of the low-resource language due to the missing correlation of hierarchical semantic information (e.g., sentence-to-word, word-to-word). Therefore, in this paper, we propose a novel multi-level alignment framework, which hierarchically learns the semantic correlation between multiple levels by leveraging well-designed alignment training tasks. In addition, we devise an attention-based fusion mechanism (AFM) to infuse semantic information from high levels. Extensive experiments on mainstream cross-lingual tasks (e.g., text classification, paraphrase identification, and named entity recognition) demonstrate the effectiveness of our proposed method, and also show that our model achieves state-of-the-art performance across various benchmarks compared to other strong baselines.
最近，跨语言迁移学习引起了学术界和工业界的广泛关注。以前的研究通常只关注基于预训练语言模型的单层次对齐（例如，词级、句子级）。然而，由于缺少层次语义信息的相关性（例如，句子到词、词到词），这导致在低资源语言的下游任务中表现不佳。因此，在本文中，我们提出了一种新颖的多层次对齐框架，通过利用精心设计的对齐训练任务，分层学习多个层次之间的语义相关性。此外，我们设计了一种基于注意力的融合机制（AFM），从高层注入语义信息。在主流的跨语言任务（例如文本分类、释义识别和命名实体识别）上的广泛实验表明了我们提出的方法的有效性，并且与其他强大的基线相比，我们的模型在各种基准测试中也达到了最先进的性能。
