Curriculum Learning for ab initio Deep Learned Refractive Optics (2302.01089v4)
Abstract: Deep optical optimization has recently emerged as a new paradigm for designing computational imaging systems using only the output image as the objective. However, it has been limited to either simple optical systems consisting of a single element such as a diffractive optical element (DOE) or metalens, or the fine-tuning of compound lenses from good initial designs. Here we present a DeepLens design method based on curriculum learning, which is able to learn optical designs of compound lenses ab initio from randomly initialized surfaces without human intervention, therefore overcoming the need for a good initial design. We demonstrate the effectiveness of our approach by fully automatically designing both classical imaging lenses and a large field-of-view extended depth-of-field computational lens in a cellphone-style form factor, with highly aspheric surfaces and a short back focal length.
- End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph., 37(4), jul 2018.
- Inference in artificial intelligence with deep optics and photonics. Nature, 588(7836):39–47, 2020.
- Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1386–1396, 2020.
- End-to-end complex lens design with differentiable ray tracing. ACM Trans. Graph, 40(4):1–13, 2021.
- dO𝑑𝑂dOitalic_d italic_O: A differentiable engine for Deep Lens design of computational imaging systems. IEEE Trans. Comput. Imaging, 2022.
- Neural nano-optics for high-quality thin lens imaging. Nat. Commun., 12(1):1–7, 2021.
- Compact snapshot hyperspectral imaging with diffracted rotation. ACM Trans. Graph., 2019.
- Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. Optica, 7(8):913–922, 2020.
- Single-shot hyperspectral-depth imaging with learned diffractive optics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2651–2660, 2021.
- Mask-ToF: Learning microlens masks for flying pixel correction in time-of-flight imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9116–9126, 2021.
- Quantization-aware deep optics for diffractive snapshot hyperspectral imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19780–19789, 2022.
- Deep optics for monocular depth estimation and 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10193–10202, 2019.
- End-to-end learned, optically coded super-resolution spad camera. ACM Transactions on Graphics (TOG), 39(2):1–14, 2020.
- Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1375–1385, 2020.
- Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In 2021 IEEE International Conference on Computational Photography (ICCP), pages 1–12. IEEE, 2021.
- Seeing through obstructions with diffractive cloaking. ACM Trans. Graph., 41(4):1–15, 2022.
- Hybrid diffractive optics design via hardware-in-the-loop methodology for achromatic extended-depth-of-field imaging. Optics Express, 30(18):32633–32649, 2022.
- Deep learning-enabled framework for automatic lens design starting point generation. Opt. Express., 29(3):3841–3854, 2021.
- Comparison of methods for end-to-end co-optimization of optical systems and image processing with commercial lens design software. Optics Express, 30(8):13556–13571, 2022.
- Revealing the preference for correcting separated aberrations in joint optic-image design. arXiv preprint arXiv:2309.04342, 2023.
- Computational optics for mobile terminals in mass production. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4245–4259, 2022.
- Warren J Smith. Modern optical engineering: the design of optical systems. McGraw-Hill Education, 2008.
- Yuke Ma et al. Design of a 16.5 megapixel camera lens for a mobile phone. Open Access library journal, 2(03):1, 2015.
- Optimization of a mobile phone camera for as-built performance. In Current Developments in Lens Design and Optical Engineering XXI, volume 11482, pages 85–94. SPIE, 2020.
- Lens design fundamentals. academic press, 2009.
- Differentiable compound optics and processing pipeline optimization for end-to-end camera design. ACM Trans. Graph., 40(2):1–19, 2021.
- The differentiable lens: Compound lens search over glass surfaces and materials for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20803–20812, 2023.
- Large depth-of-field ultra-compact microscope by progressive optimization and deep learning. Nature Communications, 14(1):4118, 2023.
- Extrapolating from lens design databases using deep learning. Optics express, 27(20):28279–28292, 2019.
- Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
- Automated curriculum learning for neural networks. In international conference on machine learning, pages 1311–1320. Pmlr, 2017.
- A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021.
- Extended depth of field through wave-front coding. Appl. Opt., 34(11):1859–1866, 1995.
- Optimized asymmetrical tangent phase mask to obtain defocus invariant modulation transfer function in incoherent imaging systems. Optics letters, 39(7):2171–2174, 2014.
- Microscope with extension of the depth of field by employing a cubic phase plate on the surface of lens. Results in Optics, 4:100107, 2021.
- Simple baselines for image restoration. arXiv preprint arXiv:2204.04676, 2022.
- U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
- Radiative backpropagation: An adjoint method for lightning-fast differentiable rendering. ACM Trans. Graph., 39, 7 2020.
- Adjoint nonlinear ray tracing. ACM Trans. Graph., 41(4):1–13, 2022.
- Path replay backpropagation: Differentiating light paths using constant memory and linear time. ACM Trans. Graph., 40:108:1–108:14, 2021.