Rethinking the Function of Neurons in KANs (2407.20667v1)
Abstract: The neurons of Kolmogorov-Arnold Networks (KANs) perform a simple summation motivated by the Kolmogorov-Arnold representation theorem, which asserts that sum is the only fundamental multivariate function. In this work, we investigate the potential for identifying an alternative multivariate function for KAN neurons that may offer increased practical utility. Our empirical research involves testing various multivariate functions in KAN neurons across a range of benchmark Machine Learning tasks. Our findings indicate that substituting the sum with the average function in KAN neurons results in significant performance enhancements compared to traditional KANs. Our study demonstrates that this minor modification contributes to the stability of training by confining the input to the spline within the effective range of the activation function. Our implementation and experiments are available at: \url{https://github.com/Ghaith81/dropkan}
- Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756, 2024.
- Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
- Surrogate-assisted genetic algorithm for wrapper feature selection. In 2021 IEEE congress on evolutionary computation (CEC), pages 776–785. IEEE, 2021.
- Ziyao Li. Kolmogorov-arnold networks are radial basis function networks. arXiv preprint arXiv:2405.06721, 2024.
- Minjong Cheon. Demonstrating the efficacy of kolmogorov-arnold networks in vision tasks. arXiv preprint arXiv:2406.14916, 2024.
- Suitability of kans for computer vision: A preliminary investigation. arXiv preprint arXiv:2406.09087, 2024.
- Kolmogorov-arnold networks (kans) for time series analysis. arXiv preprint arXiv:2405.08790, 2024.
- A benchmarking study of kolmogorov-arnold networks on tabular data. arXiv preprint arXiv:2406.14529, 2024.
- Deepokan: Deep operator network based on kolmogorov arnold networks for mechanics problems. arXiv preprint arXiv:2405.19143, 2024.
- ikan: Global incremental learning with kan for human activity recognition across heterogeneous datasets. arXiv preprint arXiv:2406.01646, 2024.
- Initial investigation of kolmogorov-arnold networks (kans) as feature extractors for imu based human activity recognition. arXiv preprint arXiv:2406.11914, 2024.
- A kan-based hybrid deep neural networks for accurate identification of transcription factor binding sites. 2024.
- Kanqas: Kolmogorov arnold network for quantum architecture search. arXiv preprint arXiv:2406.17630, 2024.
- Reduced effectiveness of kolmogorov-arnold networks on functions with noise. arXiv preprint arXiv:2407.14882, 2024.
- Exploring the limitations of kolmogorov-arnold networks in classification: Insights to software training and hardware implementation. arXiv preprint arXiv:2407.17790, 2024.
- Kan or mlp: A fairer comparison. arXiv preprint arXiv:2407.16674, 2024.
- Kagnns: Kolmogorov-arnold networks meet graph learning. arXiv preprint arXiv:2406.18380, 2024.
- Gkan: Graph kolmogorov-arnold networks. arXiv preprint arXiv:2406.06470, 2024.
- Kolmogorov-arnold graph neural networks. arXiv preprint arXiv:2406.18354, 2024.
- Fourierkan-gcf: Fourier kolmogorov-arnold network–an effective and efficient feature transformation for graph collaborative filtering. arXiv preprint arXiv:2406.01034, 2024.
- Convolutional kolmogorov-arnold networks. arXiv preprint arXiv:2406.13155, 2024.
- A temporal kolmogorov-arnold transformer for time series forecasting. arXiv preprint arXiv:2406.02486, 2024.
- Mohammed Ghaith Altarabichi. Dropkan: Regularizing kans by masking post-activations. arXiv preprint arXiv:2407.13044, 2024.
- Wav-kan: Wavelet kolmogorov-arnold networks. arXiv preprint arXiv:2405.12832, 2024.
- Seyd Teymoor Seydi. Unveiling the power of wavelets: A wavelet-based kolmogorov-arnold network for hyperspectral image classification. arXiv preprint arXiv:2406.07869, 2024.
- Hoang-Thang Ta. Bsrbf-kan: A combination of b-splines and radial basic functions in kolmogorov-arnold networks. arXiv preprint arXiv:2406.11173, 2024.
- Alireza Afzal Aghaei. fkan: Fractional kolmogorov-arnold networks with trainable jacobi basis functions. arXiv preprint arXiv:2406.07456, 2024.
- Alireza Afzal Aghaei. rkan: Rational kolmogorov-arnold networks. arXiv preprint arXiv:2406.14495, 2024.
- Sinekan: Kolmogorov-arnold networks using sinusoidal activation functions. arXiv preprint arXiv:2407.04149, 2024.