Active Perception using Neural Radiance Fields (2310.09892v2)
Abstract: We study active perception from first principles to argue that an autonomous agent performing active perception should maximize the mutual information that past observations posses about future ones. Doing so requires (a) a representation of the scene that summarizes past observations and the ability to update this representation to incorporate new observations (state estimation and mapping), (b) the ability to synthesize new observations of the scene (a generative model), and (c) the ability to select control trajectories that maximize predictive information (planning). This motivates a neural radiance field (NeRF)-like representation which captures photometric, geometric and semantic properties of the scene grounded. This representation is well-suited to synthesizing new observations from different viewpoints. And thereby, a sampling-based planner can be used to calculate the predictive information from synthetic observations along dynamically-feasible trajectories. We use active perception for exploring cluttered indoor environments and employ a notion of semantic uncertainty to check for the successful completion of an exploration task. We demonstrate these ideas via simulation in realistic 3D indoor environments.
- R. Bajcsy, Y. Aloimonos, and J. K. Tsotsos, “Revisiting active perception,” 2016.
- Y. Aloimonos, I. Weiss, and A. Bandyopadhyay, “Active vision,” International Journal of Computer Vision, vol. 1, pp. 333–356, 2004.
- J. K. Tsotsos and K. Shubina, “Attention and visual search: Active robotic vision systems that search,” 2007.
- D. Falanga, E. Mueggler, M. Faessler, and D. Scaramuzza, “Aggressive quadrotor flight through narrow gaps with onboard sensing and computing using active vision,” in ICRA, pp. 5774–5781, 2017.
- T. Novkovic, R. Pautrat, F. Furrer, M. Breyer, R. Siegwart, and J. Nieto, “Object finding in cluttered scenes using interactive perception,” 2020.
- A. Dosovitskiy and V. Koltun, “Learning to act by predicting the future,” arXiv preprint arXiv:1611.01779, 2016.
- B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in European Conference on Computer Vision, pp. 405–421, 2020.
- S. Zhi, T. Laidlow, S. Leutenegger, and A. Davison, “In-place scene labelling and understanding with implicit scene representation,” in ICCV, 2021.
- T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, pp. 102:1–102:15, July 2022.
- R. Li, H. Gao, M. Tancik, and A. Kanazawa, “NerfAcc: Efficient sampling accelerates NeRFs,” 2023.
- W. Bialek, I. Nemenman, and N. Tishby, “Predictability, complexity, and learning,” Neural computation, 2001.
- X. Pan, Z. Lai, S. Song, and G. Huang, “Activenerf: Learning where to see with uncertainty estimation,” 2022.
- E. J. Smith, M. Drozdzal, D. Nowrouzezahrai, D. Meger, and A. Romero-Soriano, “Uncertainty-driven active vision for implicit scene reconstruction,” 2022.
- B. Yamauchi, “A frontier-based approach for autonomous exploration,” in CIRA, pp. 146–151, 1997.
- D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov, “Learning to explore using active neural slam,” 2020.
- G. Kopanas and G. Drettakis, “Improving NeRF quality by progressive camera placement for navigation in complex environments,” 2023.
- P. Marza, L. Matignon, O. Simonin, D. Batra, C. Wolf, and D. S. Chaplot, “Autonerf: Training implicit scene representations with autonomous agents,” 2023.
- M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager, “Vision-only robot navigation in a neural radiance world,” 2022.
- H. Zhan, J. Zheng, Y. Xu, I. Reid, and H. Rezatofighi, “Activermap: Radiance field for active mapping and planning,” 2022.
- Y. Ran, J. Zeng, S. He, J. Chen, L. Li, Y. Chen, G. Lee, and Q. Ye, “NeurAR: Neural uncertainty for autonomous 3d reconstruction with implicit neural representations,” IEEE Robotics and Automation Letters, vol. 8, pp. 1125–1132, feb 2023.
- J. Shen, A. Ruiz, A. Agudo, and F. Moreno-Noguer, “Stochastic neural radiance fields: Quantifying uncertainty in implicit 3D representations,” 2021.
- W. Jiang, B. Lei, and K. Daniilidis, “Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information,” 2023.
- N. Sunderhauf, J. Abou-Chakra, and D. Miller, “Density-aware NeRF ensembles: Quantifying predictive uncertainty in neural radiance fields,” 2022.
- T. M. Cover, Elements of Information Theory. 1999.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017.
- J. Yu, J. E. Low, K. Nagami, and M. Schwager, “Nerfbridge: Bringing real-time, online neural radiance field training to robotics,” 2023.
- D. Mellinger and V. Kumar, “Minimum snap trajectory generation and control for quadrotors,” in ICRA, pp. 2520–2525, 2011.
- S. Folk, J. Paulos, and V. Kumar, “Rotorpy: A python-based multirotor simulator with aerodynamics for education and research,” arXiv preprint arXiv:2306.04485, 2023.
- A. Sakai, D. Ingram, J. Dinius, K. Chawla, A. Raffin, and A. Paques, “Pythonrobotics: a python code collection of robotics algorithms,” 2018.
- A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra, “Habitat 2.0: Training home assistants to rearrange their habitat,” in Advances in Neural Information Processing Systems (NeurIPS), 2021.
- M. Khanna*, Y. Mao*, H. Jiang, S. Haresh, B. Shacklett, D. Batra, A. Clegg, E. Undersander, A. X. Chang, and M. Savva, “Habitat Synthetic Scenes Dataset: An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation,” arXiv preprint, 2023.