SLACK: Stable Learning of Augmentations with Cold-start and KL regularization (2306.09998v1)
Abstract: Data augmentation is known to improve the generalization capabilities of neural networks, provided that the set of transformations is chosen with care, a selection often performed manually. Automatic data augmentation aims at automating this process. However, most recent approaches still rely on some prior information; they start from a small pool of manually-selected default transformations that are either used to pretrain the network or forced to be part of the policy learned by the automatic data augmentation algorithm. In this paper, we propose to directly learn the augmentation policy without leveraging such prior knowledge. The resulting bilevel optimization problem becomes more challenging due to the larger search space and the inherent instability of bilevel optimization algorithms. To mitigate these issues (i) we follow a successive cold-start strategy with a Kullback-Leibler regularization, and (ii) we parameterize magnitudes as continuous distributions. Our approach leads to competitive results on standard benchmarks despite a more challenging setting, and generalizes beyond natural images.
- Non-convex bilevel games with critical point selection maps. preprint arXiv:2207.04888, 2022.
- Autoaugment: Learning augmentation strategies from data. In Proc. CVPR, 2019.
- RandAugment: Practical automated data augmentation with a reduced search space. In Proc. NeurIPS, 2020.
- ImageNet: A large-scale hierarchical image database. In Proc. CVPR, 2009.
- Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.
- Michael C Fu. Gradient estimation. Handbooks in operations research and management science, 13:575–616, 2006.
- Backpropagation through the void: Optimizing control variates for black-box gradient estimation. In Proc. ICLR, 2018.
- In search of lost domain generalization. In Proc. ICLR, 2021.
- Faster AutoAugment: Learning augmentation strategies using backpropagation. In Proc. ECCV, 2020.
- Meta approach to data augmentation optimization. In Proc. WACV, 2022.
- Deep residual learning for image recognition. In Proc. CVPR, 2016.
- Population based augmentation: Efficient learning of augmentation policy schedules. In Proc. ICML, pages 2731–2741, 2019.
- Learning multiple layers of features from tiny images. 2009.
- DADA: differentiable automatic data augmentation. In Proc. ECCV, 2020.
- Fast AutoAugment. In Proc. NeurIPS, 2019.
- Online hyper-parameter learning for auto-augmentation strategy. In Proc. ICCV, 2019.
- Uniformaugment: A search-free probabilistic data augmentation approach. arXiv preprint arXiv:2003.14348, 2020.
- Direct differentiable augmentation search. In Proc. ICCV, 2021.
- TrivialAugment: tuning-free yet state-of-the-art data augmentation. In Proc. ICCV, 2021.
- Transformation pursuit for image classification. In Proc. CVPR, 2014.
- Moment matching for multi-source domain adaptation. In Proc. ICCV, pages 1406–1415, 2019.
- R Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877–898, 1976.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- Teppei Suzuki. Teachaugment: Data augmentation optimization using teacher knowledge. In Proc. CVPR, 2022.
- Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
- Addressing model vulnerability to distributional shifts over image transformation sets. In Proc. ICCV, pages 7980–7989, 2019.
- DAAS: Differentiable architecture and augmentation policy search. arXiv preprint arXiv:2109.15273, 2021.
- Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992.
- Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proc. BMVC, pages 87.1–87.12. BMVA Press, September 2016.
- Adversarial AutoAugment. In Proc. ICLR, 2020.
- Deep AutoAugmentation. In Proc. ICLR, 2022.