2000 character limit reached
Three Heads Are Better Than One: Complementary Experts for Long-Tailed Semi-supervised Learning (2312.15702v2)
Published 25 Dec 2023 in cs.CV
Abstract: We address the challenging problem of Long-Tailed Semi-Supervised Learning (LTSSL) where labeled data exhibit imbalanced class distribution and unlabeled data follow an unknown distribution. Unlike in balanced SSL, the generated pseudo-labels are skewed towards head classes, intensifying the training bias. Such a phenomenon is even amplified as more unlabeled data will be mislabeled as head classes when the class distribution of labeled and unlabeled datasets are mismatched. To solve this problem, we propose a novel method named ComPlementary Experts (CPE). Specifically, we train multiple experts to model various class distributions, each of them yielding high-quality pseudo-labels within one form of class distribution. Besides, we introduce Classwise Batch Normalization for CPE to avoid performance degradation caused by feature distribution mismatch between head and non-head classes. CPE achieves state-of-the-art performances on CIFAR-10-LT, CIFAR-100-LT, and STL-10-LT dataset benchmarks. For instance, on CIFAR-10-LT, CPE improves test accuracy by over 2.22% compared to baselines. Code is available at https://github.com/machengcheng2016/CPE-LTSSL.
- A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106: 249–259.
- Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In International Conference on Computer Vision (ICCV), 112–121.
- Learning imbalanced datasets with label-distribution-aware margin loss. Neural Information Processing Systems (NeurIPS), 32.
- SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16: 321–357.
- An embarrassingly simple baseline for imbalanced semi-supervised learning. arXiv preprint arXiv:2211.11086.
- SoftMatch: Addressing the quantity-quality tradeoff in semi-supervised Learning. In International Conference on Learning Representations (ICLR).
- An analysis of single-layer networks in unsupervised feature learning. In Artificial Intelligence and Statistics (AISTATS), 215–223. JMLR Workshop and Conference Proceedings.
- Reslt: Residual learning for long-tailed recognition. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3695–3706.
- Class-balanced loss based on effective number of samples. In Computer Vision and Pattern Recognition (CVPR), 9268–9277.
- RDA: Reciprocal distribution alignment for robust semi-supervised learning. In European Conference on Computer Vision (ECCV), 533–549. Springer.
- Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning. In Computer Vision and Pattern Recognition (CVPR), 14574–14584.
- Class-imbalanced semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning (ICML), 8082–8094. PMLR.
- Disentangling label distribution for long-tailed visual recognition. In Computer Vision and Pattern Recognition (CVPR), 6626–6636.
- Survey on deep learning with class imbalance. Journal of Big Data, 6(1): 1–54.
- Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations (ICLR).
- Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 29(8): 3573–3587.
- Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Neural Information Processing Systems (NeurIPS), 33: 14567–14579.
- Krizhevsky, A.; et al. 2009. Learning multiple layers of features from tiny images.
- Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In International Conference on Machine Learning (ICML), 11828–11843. PMLR.
- Abc: Auxiliary balanced classifier for class-imbalanced semi-supervised learning. Neural Information Processing Systems (NeurIPS), 34: 7082–7094.
- Large-scale long-tailed recognition in an open world. In Computer Vision and Pattern Recognition (CVPR), 2537–2546.
- Long-tail learning via logit adjustment. In International Conference on Learning Representations (ICLR).
- Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning. In Computer Vision and Pattern Recognition (CVPR), 9786–9796.
- A survey of semi-supervised learning methods. In Computational Intelligence and Security (CIS), volume 2, 30–34. IEEE.
- Reed, W. J. 2001. The Pareto, Zipf and other power laws. Economics letters, 74(1): 15–19.
- Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Neural Information Processing Systems (NeurIPS), 33: 596–608.
- Long-tailed recognition by routing diverse distribution-aware experts. In International Conference on Learning Representations (ICLR).
- Debiased learning from naturally imbalanced pseudo-labels. In Computer Vision and Pattern Recognition (CVPR), 14647–14657.
- FreeMatch: Self-adaptive thresholding for semi-supervised learning. In International Conference on Learning Representations (ICLR).
- Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In Computer Vision and Pattern Recognition (CVPR), 10857–10866.
- Towards realistic long-tailed semi-supervised learning: Consistency is all you need. In Computer Vision and Pattern Recognition (CVPR), 3469–3478.
- Adversarial examples improve image recognition. In Computer Vision and Pattern Recognition (CVPR), 819–828.
- Wide residual networks. In British Machine Vision Conference (BMVC). British Machine Vision Association.
- Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Neural Information Processing Systems (NeurIPS), 34: 18408–18419.
- Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Neural Information Processing Systems (NeurIPS), 35: 34077–34090.