Pseudo-label Correction for Instance-dependent Noise Using Teacher-student Framework (2311.14237v1)
Abstract: The high capacity of deep learning models to learn complex patterns poses a significant challenge when confronted with label noise. The inability to differentiate clean and noisy labels ultimately results in poor generalization. We approach this problem by reassigning the label for each image using a new teacher-student based framework termed P-LC (pseudo-label correction). Traditional teacher-student networks are composed of teacher and student classifiers for knowledge distillation. In our novel approach, we reconfigure the teacher network into a triple encoder, leveraging the triplet loss to establish a pseudo-label correction system. As the student generates pseudo labels for a set of given images, the teacher learns to choose between the initially assigned labels and the pseudo labels. Experiments on MNIST, Fashion-MNIST, and SVHN demonstrate P-LC's superior performance over existing state-of-the-art methods across all noise levels, most notably in high noise. In addition, we introduce a noise level estimation to help assess model performance and inform the need for additional data cleaning procedures.
- “Understanding deep learning (still) requires rethinking generalization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021.
- “Classification in the presence of label noise: a survey,” IEEE transactions on neural networks and learning systems, vol. 25, no. 5, pp. 845–869, 2013.
- “Training convolutional networks with noisy labels,” in 3rd International Conference on Learning Representations, ICLR 2015, 2015.
- “Online crowdsourcing: rating annotators and obtaining cost-effective labels,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. IEEE, 2010, pp. 25–32.
- “Crawling the hidden web,” in Vldb, 2001, vol. 1, pp. 129–138.
- “Learning object categories from internet image searches,” Proceedings of the IEEE, vol. 98, no. 8, pp. 1453–1466, 2010.
- Zhi-Hua Zhou, “A brief introduction to weakly supervised learning,” National science review, vol. 5, no. 1, pp. 44–53, 2018.
- “Learning from noisy examples,” Machine learning, vol. 2, pp. 343–370, 1988.
- “Part-dependent label noise: Towards instance-dependent label noise,” Advances in Neural Information Processing Systems, vol. 33, pp. 7597–7610, 2020.
- “Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 11442–11450.
- “Classification with asymmetric label noise: Consistency and maximal denoising,” in Conference on learning theory. PMLR, 2013, pp. 489–511.
- “Beyond synthetic noise: Deep learning on controlled noisy labels,” in International conference on machine learning. PMLR, 2020, pp. 4804–4815.
- “On the design of loss functions for classification: theory, robustness to outliers, and savageboost,” Advances in neural information processing systems, vol. 21, 2008.
- “Making deep neural networks robust to label noise: A loss correction approach,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1944–1952.
- “Learning from corrupted binary labels via class-probability estimation,” in International conference on machine learning. PMLR, 2015, pp. 125–134.
- “Meta label correction for learning with weak supervision,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
- “Classification with noisy labels by importance reweighting,” IEEE Transactions on pattern analysis and machine intelligence, vol. 38, no. 3, pp. 447–461, 2015.
- “Meta-weight-net: Learning an explicit mapping for sample weighting,” Advances in neural information processing systems, vol. 32, 2019.
- “Learning to reweight examples for robust deep learning,” in International conference on machine learning. PMLR, 2018, pp. 4334–4343.
- “Learning from noisy labels with distillation,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1910–1918.
- “Learning from untrusted data,” in Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, 2017, pp. 47–60.
- “Learning from noisy large-scale datasets with minimal supervision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 839–847.
- “To smooth or not? when label smoothing meets noisy labels,” arXiv preprint arXiv:2106.04149, 2021.
- “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815–823.
- “Learning to compare: Relation network for few-shot learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1199–1208.
- “Optimization as a model for few-shot learning,” in International conference on learning representations, 2016.
- “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” Advances in neural information processing systems, vol. 31, 2018.
- “Combating label noise in deep learning using abstention,” in International Conference on Machine Learning. PMLR, 2019, pp. 6234–6243.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.