Kim Kyung Geun, Lee Byeong Tak
VUNO Inc., Seoul, Republic of Korea.
Medical AI Co., Ltd., Seoul, Republic of Korea.
Front Artif Intell. 2024 Aug 6;7:1397298. doi: 10.3389/frai.2024.1397298. eCollection 2024.
Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.
自然界中许多不同的现象往往内在地编码了短期和长期的时间依赖性，这尤其源于时间流动的方向。在这方面，我们发现实验证据表明，对于时间戳越接近的这些事件，其发生率越高。然而，为了使基于注意力的模型能够学习短期依赖性中的这些规律，需要大量数据，而这通常是不可行的。这是因为，虽然基于注意力的模型擅长学习分段时间依赖性，但它们缺乏对时间序列中的偏差进行编码的结构。作为一种解决方案，我们提出了一种简单有效的方法，通过将可学习的自适应内核直接应用于注意力矩阵，使注意力层能够更好地编码这些数据集的短期时间偏差。我们选择使用电子健康记录（EHR）数据集进行各种预测任务的实验，因为它们是具有潜在长期和短期时间依赖性的很好例子。与大多数任务和数据集上表现最佳的模型相比，我们的实验显示出了卓越的分类结果。
