Reinhardt Eric, Ramakrishnan Dinesh, Gleyzer Sergei
Department of Physics and Astronomy, The University of Alabama, Tuscaloosa, AL, United States.
Front Artif Intell. 2025 Jan 15;7:1462952. doi: 10.3389/frai.2024.1462952. eCollection 2024.
Recent work has established an alternative to traditional multi-layer perceptron neural networks in the form of Kolmogorov-Arnold Networks (KAN). The general KAN framework uses learnable activation functions on the edges of the computational graph followed by summation on nodes. The learnable edge activation functions in the original implementation are basis spline functions (B-Spline). Here, we present a model in which learnable grids of B-Spline activation functions are replaced by grids of re-weighted sine functions (SineKAN). We evaluate numerical performance of our model on a benchmark vision task. We show that our model can perform better than or comparable to B-Spline KAN models and an alternative KAN implementation based on periodic cosine and sine functions representing a Fourier Series. Further, we show that SineKAN has numerical accuracy that could scale comparably to dense neural networks (DNNs). Compared to the two baseline KAN models, SineKAN achieves a substantial speed increase at all hidden layer sizes, batch sizes, and depths. Current advantage of DNNs due to hardware and software optimizations are discussed along with theoretical scaling. Additionally, properties of SineKAN compared to other KAN implementations and current limitations are also discussed.
最近的研究工作已经建立了一种以柯尔莫哥洛夫 - 阿诺德网络（KAN）形式替代传统多层感知器神经网络的方法。一般的KAN框架在计算图的边上使用可学习的激活函数，然后在节点上进行求和。原始实现中的可学习边激活函数是基样条函数（B样条）。在这里，我们提出了一个模型，其中B样条激活函数的可学习网格被重新加权的正弦函数网格（SineKAN）所取代。我们在一个基准视觉任务上评估了我们模型的数值性能。我们表明，我们的模型可以比B样条KAN模型以及基于表示傅里叶级数的周期余弦和正弦函数的另一种KAN实现表现得更好或相当。此外，我们表明SineKAN具有与密集神经网络（DNN）相当的数值精度。与两个基线KAN模型相比，SineKAN在所有隐藏层大小、批量大小和深度上都实现了显著的速度提升。讨论了由于硬件和软件优化导致的DNN的当前优势以及理论缩放。此外，还讨论了SineKAN与其他KAN实现相比的特性以及当前的局限性。
