A study on effectiveness of extreme learning machine (1409.3924v1)

Abstract: Extreme learning machine (ELM), proposed by Huang et al., has been shown a promising learning algorithm for single-hidden layer feedforward neural networks (SLFNs). Nevertheless, because of the random choice of input weights and biases, the ELM algorithm sometimes makes the hidden layer output matrix H of SLFN not full column rank, which lowers the effectiveness of ELM. This paper discusses the effectiveness of ELM and proposes an improved algorithm called EELM that makes a proper selection of the input weights and bias before calculating the output weights, which ensures the full column rank of H in theory. This improves to some extend the learning rate (testing accuracy, prediction accuracy, learning time) and the robustness property of the networks. The experimental results based on both the benchmark function approximation and real-world problems including classification and regression applications show the good performances of EELM.

• The paper introduces Effective Extreme Learning Machine (EELM) to enhance SLFN performance by ensuring the hidden layer output matrix has full column rank.
• The methodology employs deterministic selection of input weights and biases with Gaussian radial basis functions and a sorted sample approach to improve learning speed.
• Empirical validation on benchmark regression and real-world datasets demonstrates significant reductions in training and testing errors compared to standard ELM.

Overview of "A Study on Effectiveness of Extreme Learning Machine"

The paper "A Study on Effectiveness of Extreme Learning Machine" by Yuguang Wang, Feilong Cao, and Yubo Yuan focuses on assessing the capabilities of the Extreme Learning Machine (ELM) algorithm for single-hidden layer feedforward neural networks (SLFNs). This algorithm, introduced by Huang et al., has been acknowledged for its rapid learning in training SLFNs, which are known for their potential in approximating complex nonlinear mappings. The authors offer a critique of the existing ELM methodology and introduce an Enhanced Extreme Learning Machine (EELM) that seeks to overcome identified limitations, specifically regarding the randomness in selection of input weights and biases, which can lead to an ineffective learning setup.

Contributions of the Paper

The central contribution of this work is the introduction of the Effective Extreme Learning Machine (EELM), an enhancement over the standard ELM. The EELM algorithm aims to ensure that the hidden layer output matrix achieves full column rank, thereby improving the robustness and learning speed of the SLFN. The modified algorithm strategically pre-processes input weights and biases to address potential deficiencies associated with random initialization. This adjustment theoretically guarantees the full column rank property of the hidden layer output matrix, promoting improvements in training accuracy, testing accuracy, and learning speed.

Core Algorithmic Advancements

The proposed EELM algorithm leverages:

1. Proper Selection of Input Weights and Biases: By ensuring a full column rank of the hidden layer output matrix through appropriate pre-selection techniques.
2. Use of Gaussian Radial Basis Activation Functions: These are selected for their suitability in maintaining non-singular matrices.
3. Sorted Sample Approach: An innovative sorting technique based on affine transformation that enhances network architecture.

Empirical Validation

The paper provides thorough empirical validation of the EELM methodology via benchmark regression and real-world classification/regression tasks. Results indicate a substantial decrease in both training and testing errors when using EELM compared to standard ELM. For example, the EELM demonstrated a notable prediction accuracy improvement in the modeling of the 'SinC' function, with significant gains in prediction robustness over broader ranges of input data. Additionally, in real-world tests using datasets such as Diabetes and Statlog, EELM outperformed typical ELM implementations in several cases, particularly concerning robustness in regression scenarios.

Implications and Future Directions

The findings imply that careful design of input layer parameters in neural networks can mitigate some inherent inefficiencies observed with purely randomized methodologies like ELM. Practically, this research suggests that using deterministic preprocessing steps in network initialization is vital for tasks requiring high precision and robustness, especially in regression contexts.

Theoretical enhancements also suggest further exploration into deterministic parameterization methods for different types of activation functions, optimizing network initialization strategies irrespective of network depth or input dimensionality.

Future research could extend the principles illustrated in EELM to multi-layer architectures, examining the scalability of these strategies and verifying applicability across diverse application domains, such as time-series forecasting and real-time data inference, which demand rapid and precise learning capabilities.