Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion (1902.03680v3)

Abstract: The predictive performance of supervised learning algorithms depends on the quality of labels. In a typical label collection process, multiple annotators provide subjective noisy estimates of the "truth" under the influence of their varying skill-levels and biases. Blindly treating these noisy labels as the ground truth limits the accuracy of learning algorithms in the presence of strong disagreement. This problem is critical for applications in domains such as medical imaging where both the annotation cost and inter-observer variability are high. In this work, we present a method for simultaneously learning the individual annotator model and the underlying true label distribution, using only noisy observations. Each annotator is modeled by a confusion matrix that is jointly estimated along with the classifier predictions. We propose to add a regularization term to the loss function that encourages convergence to the true annotator confusion matrix. We provide a theoretical argument as to how the regularization is essential to our approach both for the case of single annotator and multiple annotators. Despite the simplicity of the idea, experiments on image classification tasks with both simulated and real labels show that our method either outperforms or performs on par with the state-of-the-art methods and is capable of estimating the skills of annotators even with a single label available per image.

• The paper proposes a novel regularization method that jointly learns annotator confusion matrices and true label distributions from noisy data.
• It integrates trace norm regularization within the loss function to enforce accurate modeling of annotator biases and errors.
• Empirical results on datasets like MNIST and cardiac ultrasound show improved classification accuracy in sparse annotation scenarios.

Summary of "Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion"

The paper "Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion" presents a novel approach to enhancing the predictive performance of supervised learning models trained on data labeled with noisy annotations. Specifically, the authors address the challenge where labels are provided by multiple annotators of varying skill levels and biases, a common scenario in fields such as medical imaging. The approach aims to jointly learn the expertise and biases of individual annotators and recover the true label distribution from these confounded observations.

The core contribution is the integration of a regularization term into the loss function, which promotes accurate estimation of annotator confusion matrices. This regularization encourages annotator models to be as unreliable as possible while ensuring they still align with observed data. This method contrasts with previous approaches that often applied expectation-maximization (EM) algorithms, which could be computationally intensive or impractical with sparse label scenarios (e.g., where each image is labeled by only one annotator).

Key Methodological Insights

1. Probabilistic Model of Noisy Labels: The paper models each annotator's labeling behavior using confusion matrices, which capture the probability of annotators assigning incorrect labels. The authors assume that annotators are independent and that label noise is image-independent. This simplifies the joint probability of observing noisy labels into a product of individual probabilities conditioned by the true label distribution.
2. Regularization with Trace Norm: By adding a trace regularization to the cross-entropy loss, the method explicitly penalizes configurations where the average confusion matrix deviates from the identity matrix, thereby enforcing more accurate modeling of individual annotators' biases and errors. The theoretical results underpin the capacity of this approach to recover annotation noise as long as the aggregate confusion matrix is diagonally dominant.
3. Empirical Validation: The authors demonstrate empirical success on datasets including MNIST and CIFAR-10 with simulated annotators of diverse skill levels and confusion patterns. The proposed method outperforms existing state-of-the-art methods, notably in scenarios where only one label per image is available, showcasing robustness to label sparsity.
4. Real-World Applications: Application to a challenging real-world problem of cardiac view classification using ultrasound images, labeled by annotators with varying expertise, highlights the method's practical capabilities. The model not only improved classification accuracy but also provided insights into annotator skill variability.

Implications and Future Developments

This work has significant implications for supervised learning in domains where annotations are costly and expertise levels vary. The ability to more accurately model annotator error and improve true label estimation can lead to more reliable models without the necessity of obtaining costly multiple annotations per sample.

For future developments, several extensions to this framework could be considered:

• Scalability to Massive Label Spaces: Imposing structures such as low-rank approximations on the confusion matrices could extend applicability to large-scale problems with extensive class sets.
• Relaxing Image-Independence Assumptions: Incorporating input-dependent noise modeling could address scenarios where label ambiguity is inherently tied to challenging inputs.

In conclusion, the paper presents a practical and theoretically sound contribution to the field of learning from noisy data, particularly valuable in applications like medical imaging, where label quality significantly influences model performance.