Developmental Pretraining (DPT) for Image Classification Networks (2312.00304v1)
Abstract: In the backdrop of increasing data requirements of Deep Neural Networks for object recognition that is growing more untenable by the day, we present Developmental PreTraining (DPT) as a possible solution. DPT is designed as a curriculum-based pre-training approach designed to rival traditional pre-training techniques that are data-hungry. These training approaches also introduce unnecessary features that could be misleading when the network is employed in a downstream classification task where the data is sufficiently different from the pre-training data and is scarce. We design the curriculum for DPT by drawing inspiration from human infant visual development. DPT employs a phased approach where carefully-selected primitive and universal features like edges and shapes are taught to the network participating in our pre-training regime. A model that underwent the DPT regime is tested against models with randomised weights to evaluate the viability of DPT.
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
- Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions. Journal of big Data, 8:1–74, 2021.
- A study of cnn and transfer learning in medical imaging: Advantages, challenges, future scope. Sustainability, 15(7):5930, 2023.
- A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76, 2021.
- Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
- A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021.
- Baby steps: How “less is more” in unsupervised dependency parsing. 2009.
- Self-paced curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
- Lessons from infant learning for unsupervised machine learning. Nature Machine Intelligence, 4(6):510–520, 2022.
- A developmental approach to machine learning? Frontiers in psychology, 8:2124, 2017.
- Recurrent neural circuits for contour detection. arXiv preprint arXiv:2010.15314, 2020.
- Potential downside of high initial visual acuity. Proceedings of the National Academy of Sciences, 115(44):11333–11338, 2018.
- Dense extreme inception network for edge detection. Pattern Recognition, 139:109461, 2023.
- A El Korchi. 2d geometric shapes dataset. Mendeley Data, 1:2020, 2020.
- Jeremy Howard. imagenette.
- Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
- Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
- Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67–74. IEEE, 2018.