Versatile Navigation under Partial Observability via Value-guided Diffusion Policy (2404.02176v1)
Abstract: Route planning for navigation under partial observability plays a crucial role in modern robotics and autonomous driving. Existing route planning approaches can be categorized into two main classes: traditional autoregressive and diffusion-based methods. The former often fails due to its myopic nature, while the latter either assumes full observability or struggles to adapt to unfamiliar scenarios, due to strong couplings with behavior cloning from experts. To address these deficiencies, we propose a versatile diffusion-based approach for both 2D and 3D route planning under partial observability. Specifically, our value-guided diffusion policy first generates plans to predict actions across various timesteps, providing ample foresight to the planning. It then employs a differentiable planner with state estimations to derive a value function, directing the agent's exploration and goal-seeking behaviors without seeking experts while explicitly addressing partial observability. During inference, our policy is further enhanced by a best-plan-selection strategy, substantially boosting the planning success rate. Moreover, we propose projecting point clouds, derived from RGB-D inputs, onto 2D grid-based bird-eye-view maps via semantic segmentation, generalizing to 3D environments. This simple yet effective adaption enables zero-shot transfer from 2D-trained policy to 3D, cutting across the laborious training for 3D policy, and thus certifying our versatility. Experimental results demonstrate our superior performance, particularly in navigating situations beyond expert demonstrations, surpassing state-of-the-art autoregressive and diffusion-based baselines for both 2D and 3D scenarios.
- Is conditional generative modeling all you need for decision making? In ICLR, 2023.
- Offline reinforcement learning via high-fidelity generative behavior modeling. In ICLR, 2023.
- Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv:2208.04202, 2022.
- Diffusion policy: Visuomotor policy learning via action diffusion. In RSS, 2023.
- Pointvector: A vector representation in point cloud analysis. In CVPR, 2023.
- Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
- Implicit generation and generalization in energy-based models. arXiv:1903.08689, 2019.
- Compositional visual generation with energy based models. In NeurIPS, 2020.
- Denoising diffusion probabilistic models. In NeurIPS, 2020.
- Estimation of non-normalized statistical models by score matching. JMLR, 6(4):695–709, 2005.
- Towards real-world navigation with deep differentiable planners. In CVPR, 2022.
- Planning with diffusion for flexible behavior synthesis. In ICML, 2022.
- Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998.
- Qmdp-net: Deep learning for planning under partial observability. In NeurIPS, 2017.
- Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022.
- Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv:2005.01643, 2020.
- Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In ICML, 2023.
- Learning policies for partially observable environments: Scaling up. In ICML, 1995.
- Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In CoRL, 2022.
- Receding horizon control of nonlinear systems. In CDC, 1988.
- Improved denoising diffusion probabilistic models. In ICML, 2021.
- Capability iteration network for robot path planning. IJRA, 37(3):266–272, 2022.
- Learning non-convergent non-persistent short-run mcmc toward energy-based model. In NeurIPS, 2019.
- Imitating human behaviour with diffusion models. In ICLR, 2023.
- Joelle Pineau. Tractable planning under uncertainty: exploiting structure. Carnegie Mellon University, 2004.
- Value iteration networks on multiple levels of abstraction. In RSS, 2019.
- Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
- Edward J Sondik. The optimal control of partially observable markov decision processes. PhD thesis, Stanford University, 1971.
- Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
- Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
- Value iteration networks. In NeurIPS, 2016.
- Diffusebot: Breeding soft robots with physics-augmented generative diffusion models. In NeurIPS, 2023a.
- Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR, 2023b.
- Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
- David Bruce Wilson. Generating random spanning trees more quickly than the cover time. In STOC, 1996.
- Swin3d: A pretrained transformer backbone for 3d indoor scene understanding. arXiv:2304.06906, 2023.