SELF: Learning to Filter Noisy Labels with Self-Ensembling (1910.01842v1)

Abstract: Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

• The paper introduces a framework that iteratively filters noisy labels using self-ensemble predictions to improve DNN performance.
• It leverages a two-step process combining semi-supervised learning and dynamic label filtering, yielding significant accuracy gains on benchmarks.
• Experimental results on CIFAR-10, CIFAR-100, and ImageNet show that SELF robustly maintains classification accuracy even at high noise levels.

Insightful Overview of "SELF: Learning to Filter Noisy Labels with Self-Ensembling"

The paper "SELF: Learning to Filter Noisy Labels with Self-Ensembling" introduces a novel framework for improving the generalization performance of Deep Neural Networks (DNNs) when trained on datasets with noisy labels. The method, termed self-ensemble label filtering (SELF), focuses on progressively filtering out noisy labels during the training process, thus allowing the network to learn primarily from clean data. This approach effectively addresses a critical challenge in supervised learning—label noise—by exploiting the ensemble of model predictions accumulated across various training epochs.

Core Methodology

The framework fundamentally relies on a two-step iterative process: (1) training with semi-supervised learning objectives and (2) filtering potentially noisy labels. SELF leverages a self-supervised mechanism, where it collects running averages of predictions (self-ensemble predictions) over multiple training epochs. This ensemble of predictions helps in identifying consistent patterns over time, distinguishing noisy labels from clean ones. Specifically, these self-ensemble predictions allow the model to focus on correct samples by removing or down-weighting noisy ones from the supervised learning process.

Notably, the approach does not discard the noisy labels entirely. Instead, SELF dynamically utilizes these samples through semi-supervised learning objectives within the unsupervised loss function. Such integration ensures the model benefits from the entire dataset, albeit with different supervision strengths, enhancing robustness against noise.

Technical Efficacy and Experimental Results

The robustness and efficacy of the SELF framework are demonstrated across various image classification datasets, including CIFAR-10, CIFAR-100, and ImageNet. Experiments under both symmetric and asymmetric noise scenarios reveal substantial performance improvements over existing noise-aware techniques. For instance, SELF achieves significant gains in classification accuracy, particularly at high noise rates like 80%, where many models typically falter.

Specific results highlight that SELF outperforms prior works in terms of accuracy, maintaining higher robust classification accuracy across different noise levels as showcased in benchmark datasets like CIFAR-10 and CIFAR-100. Moreover, the method exhibits marked resiliency to the architectural choices of the neural networks, indicating its general applicability and adaptability.

Implications and Future Directions

The implications of this research are twofold—practical and theoretical. Practically, the SELF framework provides a scalable and adaptable solution to enhance the reliability of machine learning models in real-world applications, where data might often be noisy or imperfectly labeled. Theoretically, the notion of utilizing self-ensemble predictions to guide iterative filtering opens novel avenues for research in self-supervised learning paradigms, potentially extending beyond the domain of noise-aware learning to other challenging contexts within AI.

Looking ahead, future developments could explore more sophisticated ensemble methods or integrate additional external data sources to further stabilize and refine the filtering process. There is also potential to adapt the SELF framework for more complex domains like natural language processing or time-series forecasting, broadening its applicability within the AI landscape. Integrating advanced uncertainty estimation techniques could also refine the filtering mechanism, further reducing the influence of label noise on the learning process.

In conclusion, the SELF framework represents a significant methodological advancement in the domain of noise-tolerant learning. By focusing on self-ensemble predictions and leveraging semi-supervised learning, it provides a robust approach to mitigate the adverse impacts of noisy labels, setting a foundation for further innovations in this critical area of machine learning research.