KAN or MLP: A Fairer Comparison

Comparative Analysis of KAN and MLP Architectures

The paper "KAN or MLP: A Fairer Comparison" authored by Runpeng Yu, Weihao Yu, and Xinchao Wang from the National University of Singapore, offers an in-depth comparative analysis between Kolmogorov–Arnold Networks (KAN) and Multi-Layer Perceptrons (MLP) under controlled experimental conditions. This work does not introduce a novel method but rather ensures a rigorous comparative framework by aligning the number of parameters and FLOPs in the two architectures. The comprehensive evaluation spans across various domains, including ML, computer vision (CV), NLP, audio processing, and symbolic formula representation.

Key Contributions and Findings

• Controlled Comparison: The study meticulously balances KAN and MLP architectures by aligning their parameters and FLOPs to provide a fair comparison. This approach addresses the deficiencies of previous studies that lacked such stringent control.
• Task Performance: The results reveal that MLP generally outperforms KAN across most tasks except symbolic formula representation. Specifically:
• Machine Learning: MLP maintains a competitive edge over KAN in 6 out of 8 datasets tested.
• Computer Vision: MLP consistently surpasses KAN across all CV datasets.
• Natural Language and Audio Processing: MLP demonstrates superior performance in both NLP and audio tasks.
• Symbolic Formula Representation: KAN shows a clear advantage owing to its B-spline activation functions.
• Ablation Studies: The study further validates that KAN's advantage in symbolic formula representation is primarily derived from the use of B-spline activation functions. When MLP is equipped with B-spline activation, it matches or surpasses KAN's performance in these tasks, whereas for other domains, the use of B-spline in MLP yields negligible improvements.
• Continual Learning: Contrary to earlier findings, the study shows that KAN's forgetting issue is more pronounced than MLP in standard class-incremental continual learning settings, disputing claims of superior KAN performance in such tasks.

Technical Insights

• Model Formulations: The paper details the mathematical formulations and forward equations of KAN and MLP. KAN employs learnable B-spline functions on network edges, while MLP traditionally uses fixed activation functions.
• Parametric Analysis: The paper provides explicit formulas for computing the number of parameters and FLOPs for both KAN and MLP, ensuring precise control in the comparative studies. This mathematical rigor substantiates the paper's claims regarding the fair evaluation of the two models.
• Architecture Details: Insights into how varying the activation functions and their positions (before or after linear transformations) impact performance across different tasks are elucidated, indicating that the choice of activation functions significantly affects the networks' suitability for various tasks.

Practical and Theoretical Implications

• Practical Applications: For researchers and practitioners, this study serves as a crucial reference for selecting between KAN and MLP architectures depending on the task at hand. Specific preference for symbolic formula representation should lean towards KAN, while other practical applications including ML, CV, NLP, and audio processing are better suited for MLP.
• Future Research: The findings invite further exploration into advanced activation functions and their integration within MLP-like structures. The results also call for re-evaluation of KAN's applicability in continual learning scenarios, which might lead to development of hybrid models leveraging MLP's strengths with innovative activation functions.

Conclusion

This comparative study provides a methodologically robust framework for evaluating KAN against MLP, offering critical insights into their performance across diverse tasks. By controlling for parameters and FLOPs, the paper brings forth objective evidence highlighting MLP's superior versatility except in symbolic formula representation tasks where KAN takes precedence. These results not only clarify the functional distinctions between these architectures but also pave the way for future advancements in neural network design and application.