Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Active Learning: A Reality Check (2403.14800v1)

Published 21 Mar 2024 in cs.LG, cs.AI, and cs.CV

Abstract: We conduct a comprehensive evaluation of state-of-the-art deep active learning methods. Surprisingly, under general settings, no single-model method decisively outperforms entropy-based active learning, and some even fall short of random sampling. We delve into overlooked aspects like starting budget, budget step, and pretraining's impact, revealing their significance in achieving superior results. Additionally, we extend our evaluation to other tasks, exploring the active learning effectiveness in combination with semi-supervised learning, and object detection. Our experiments provide valuable insights and concrete recommendations for future active learning studies. By uncovering the limitations of current methods and understanding the impact of different experimental settings, we aim to inspire more efficient training of deep learning models in real-world scenarios with limited annotation budgets. This work contributes to advancing active learning's efficacy in deep learning and empowers researchers to make informed decisions when applying active learning to their tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Active learning for deep detection neural networks. In ICCV, 2019.
  2. The power of ensembles for active learning in image classification. In CVPR, 2018.
  3. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In ICLR.
  4. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
  5. Adamatch: A unified approach to semi-supervised learning and domain adaptation. CoRR, abs/2106.04732, 2021.
  6. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
  7. Active learning for deep object detection via probabilistic modeling. In ICCV, 2021.
  8. Not all labels are equal: Rationalizing the labeling costs for training object detection. In CVPR, 2022.
  9. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
  10. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1183–1192, 2017.
  11. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In ECCV, 2020.
  12. Caltech-256 object category dataset. 2007.
  13. Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.
  14. Scalable active learning for object detection. In IV, 2020.
  15. Deep residual learning for image recognition. In CVPR, 2016.
  16. Bayesian active learning for classification and preference learning. CoRR, abs/1112.5745, 2011.
  17. Combining generative and discriminative models for semantic segmentation of CT scans via active learning. In Information Processing in Medical Imaging, 2011.
  18. Consistency-based semi-supervised learning for object detection. In NeurIPS, 2019.
  19. Learning visual features from large weakly supervised data. In ECCV, 2016.
  20. Localization-aware active learning for object detection. In ACCV, 2018.
  21. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
  22. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems 32, pages 7024–7035, 2019.
  23. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
  24. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, 2017.
  25. Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop on Challenges in Representation Learning, 2013.
  26. Heterogeneous uncertainty sampling for supervised learning. In Machine Learning, Proceedings of the Eleventh International Conference, pages 148–156, 1994.
  27. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pages 3–12. ACM/Springer, 1994.
  28. Microsoft COCO: common objects in context. In European Conference in Computer Vision, pages 740–755, 2014.
  29. Ssd: Single shot multibox detector. In European Conference on Computer Vision (ECCV), 2016.
  30. Latent structured active learning. In Advances in Neural Information Processing Systems, pages 728–736, 2013.
  31. Exploring the limits of weakly supervised pretraining. In European Conference in Computer Vision, pages 185–201, 2018.
  32. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 41(8):1979–1993, 2019.
  33. A metric learning reality check. In ECCV, 2020.
  34. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pages 3239–3250, 2018.
  35. George Orwell. Animal farm. In William Collins, 1946.
  36. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, pages 3546–3554, 2015.
  37. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91–99, 2015.
  38. Margin-based active learning for structured output spaces. In European Conference on Machine Learning, pages 413–424, 2006.
  39. Deep active learning for object detection. In BMVC, 2018.
  40. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
  41. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
  42. Burr Settles. Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2012.
  43. An analysis of active learning strategies for sequence labeling tasks. In Empirical Methods in Natural Language Processing, pages 1070–1079, 2008.
  44. Query by committee. In Conference on Computational Learning Theory, pages 287–294, 1992.
  45. Claude E. Shannon. A mathematical theory of communication. Mobile Computing and Communications Review, 5(1):3–55, 2001.
  46. Variational adversarial active learning. CoRR, abs/1904.00370, 2019.
  47. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.
  48. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pages 1195–1204, 2017.
  49. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, 2:45–66, 2001.
  50. Bayesian generative active deep learning. In ICML, 2019.
  51. Large-scale live active learning: Training object detectors with crawled data and crowds. In Computer Vision and Pattern Recognition, pages 1449–1456, 2011.
  52. Learning loss for active learning. In Computer Vision and Pattern Recognition, pages 93–102. Computer Vision Foundation / IEEE, 2019.
  53. Multiple instance active learning for object detection. In CVPR, 2021.
  54. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In NeurIPS, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets