A survey on modern trainable activation functions (2005.00817v4)

Abstract: In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.

• The paper presents a comprehensive taxonomy that categorizes activation functions into fixed and trainable types, illuminating design choices in neural networks.
• It details various trainable functions such as Parametric ReLU and Swish, explaining their mathematical formulations and gradient-based optimization processes.
• The study highlights performance gains achieved with adaptable activations while addressing the challenge of increased computational complexity in deep learning architectures.

A Survey on Modern Trainable Activation Functions

The paper "A Survey on Modern Trainable Activation Functions" by Apicella et al. serves as a comprehensive review of activation functions in neural networks, with a focus on those that are adaptable, learnable, or trainable. The main objective of the paper is to examine the landscape of activation functions, especially those with properties that can be adjusted during the training process to improve neural network performance.

The authors initiate the discourse by revisiting the concept of activation functions, discussing their role in artificial neural networks, and presenting a historical overview. They highlight how activation functions like ReLU, Sigmoid, and Tanh have significantly improved neural network performance by addressing issues like vanishing gradients. The introduction of trainable activation functions introduces a level of adaptability into the architecture, potentially enhancing network performance further.

The taxonomy presented forms the cornerstone of the paper's contributions, organizing activation functions into fixed-shape and trainable categories. Fixed-shape functions are separated into "classic" and "rectified-based" functions, with examples including the traditional sigmoid and the more modern ReLU family. Trainable activation functions are divided into parameterized standard functions and ensemble methods, with the latter further subdivided into linear combinations of one-to-one functions. This taxonomy offers clarity and structure, enabling an easier comparison and classification of existing functions.

A notable segment of the paper is the discussion on trainable activation functions, where the authors explore how modifications in the architecture can lead to different dynamics in training and performance. The text describes in detail various trainable activation functions such as Adjustable Generalized Sigmoid, Parametric ReLU, and Swish, elucidating their formation and highlighting the tunable parameters that set them apart from their fixed counterparts. Each function's mathematical formulation and adaptation process through gradient-based training methodologies are thoroughly analyzed.

Key numerical results are explored, illustrating how trainable activation functions often achieve superior performance metrics compared to non-trainable ones. Nonetheless, the paper stresses that the performance gains may not solely be attributed to the adaptability of the activation function but may also arise from the increased complexity of the architecture or the right tuning of other hyperparameters.

The authors provide evidence that numerous trainable activation functions can be effectively modeled as sub-networks composed of simpler, non-trainable functions, further providing the insight that similar expressivity can be achieved by employing deeper networks with classic activation functions and additional constraints. This perspective opens up intriguing possibilities for network design, suggesting that performance improvements can be obtained through judicious architectural decisions without necessitating complex activation functions.

Despite the diverse approaches to trainable activation functions, the paper identifies a shared limitation: the task of effectively integrating these functions into existing architectures without markedly increasing computational complexity remains a challenge. By documenting the trade-offs between model complexity and performance, the paper provides a valuable resource for designing training algorithms that incorporate trainable activation functions without incurring undue computational overhead.

In conclusion, this survey elucidates the landscape of trainable activation functions, providing a structured taxonomy and deep insights into their mechanisms and impacts on network performance. The findings presented underscore the potential of trainable activation functions to enhance network learning, while also recognizing the inherent complexities and challenges involved. Looking forward, these insights may galvanize further research in optimizing trainable activation functions for various applications, potentially heralding new design paradigms in deep learning architectures.