Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Factorized Motion Fields for Fast Sparse Input Dynamic View Synthesis (2404.11669v3)

Published 17 Apr 2024 in cs.CV

Abstract: Designing a 3D representation of a dynamic scene for fast optimization and rendering is a challenging task. While recent explicit representations enable fast learning and rendering of dynamic radiance fields, they require a dense set of input viewpoints. In this work, we focus on learning a fast representation for dynamic radiance fields with sparse input viewpoints. However, the optimization with sparse input is under-constrained and necessitates the use of motion priors to constrain the learning. Existing fast dynamic scene models do not explicitly model the motion, making them difficult to be constrained with motion priors. We design an explicit motion model as a factorized 4D representation that is fast and can exploit the spatio-temporal correlation of the motion field. We then introduce reliable flow priors including a combination of sparse flow priors across cameras and dense flow priors within cameras to regularize our motion model. Our model is fast, compact and achieves very good performance on popular multi-view dynamic scene datasets with sparse input viewpoints. The source code for our model can be found on our project page: https://nagabhushansn95.github.io/publications/2024/RF-DeRF.html.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  2. Ang Cao and Justin Johnson. 2023. Hexplane: A Fast Representation for Dynamic Scenes. arXiv e-prints, Article arXiv:2301.09632 (2023), arXiv:2301.09632 pages. arXiv:2301.09632
  3. TensoRF: Tensorial Radiance Fields. In Proceedings of the European Conference on Computer Vision (ECCV).
  4. Depth-Supervised NeRF: Fewer Views and Faster Training for Free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  5. Fast Dynamic Radiance Fields with Time-Aware Neural Voxels. In Proceedings of the SIGGRAPH Asia 2022 Conference Papers. https://doi.org/10.1145/3550469.3555383
  6. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. arXiv e-prints, Article arXiv:2301.10241 (2023), arXiv:2301.10241 pages. arXiv:2301.10241
  7. Plenoxels: Radiance Fields Without Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  8. Forward Flow for Novel View Synthesis of Dynamic Scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  9. Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  10. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics (TOG) 42, 4 (2023).
  11. InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  12. Fast View Synthesis of Casual Videos. arXiv e-prints, Article arXiv:2312.02135 (2023), arXiv:2312.02135 pages. arXiv:2312.02135
  13. Neural 3D Video Synthesis From Multi-View Video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  14. Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  15. Learning by Analogy: Reliable Supervision From Transformations for Unsupervised Optical Flow Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  16. David G Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision (IJCV) 60 (2004), 91–110.
  17. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proceedings of the European Conference on Computer Vision (ECCV).
  18. SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting With Neural Radiance Fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  19. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics (ToG) 41, 4 (2022), 1–15.
  20. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis From Sparse Inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  21. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  22. R2D2: Reliable and Repeatable Detector and Descriptor. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
  23. Dense Depth Priors for Neural Radiance Fields From Sparse Input Views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  24. Dataset and Pipeline for Multi-View Light-Field Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop.
  25. Johannes L. Schonberger and Jan-Michael Frahm. 2016. Structure-From-Motion Revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  26. SWAGS: Sampling Windows Adaptively for Dynamic 3D Gaussian Splatting. arXiv e-prints, Article arXiv:2312.13308 (2023), arXiv:2312.13308 pages. arXiv:2312.13308
  27. ZeroRF: Fast Sparse View 360° Reconstruction with Zero Pretraining. arXiv e-prints, Article arXiv:2312.09249 (2023), arXiv:2312.09249 pages. arXiv:2312.09249
  28. SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions. In Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques - Asia (SIGGRAPH-Asia).
  29. Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR). https://doi.org/10.1109/ISMAR55827.2022.00100
  30. Nagabhushan Somraj and Rajiv Soundararajan. 2023. ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields. In Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH). https://doi.org/10.1145/3588432.3591539
  31. Direct Voxel Grid Optimization: Super-Fast Convergence for Radiance Fields Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  32. Zachary Teed and Jia Deng. 2020. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In Proceedings of the European Conference on Computer Vision (ECCV).
  33. SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates. (June 2023).
  34. Flow Supervision for Deformable NeRF. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  35. Multiscale structural similarity for image quality assessment. In Proceedings of the Asilomar Conference on Signals, Systems Computers.
  36. 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. arXiv e-prints, Article arXiv:2310.08528 (2023), arXiv:2310.08528 pages. arXiv:2310.08528
  37. ReconFusion: 3D Reconstruction with Diffusion Priors. arXiv e-prints, Article arXiv:2312.02981 (2023), arXiv:2312.02981 pages. arXiv:2312.02981
  38. Jamie Wynn and Daniyar Turmukhambetov. 2023. DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models. arXiv e-prints, Article arXiv:2302.12231 (2023), arXiv:2302.12231 pages. arXiv:2302.12231
  39. SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting. arXiv e-prints, Article arXiv:2312.00206 (2023), arXiv:2312.00206 pages. arXiv:2312.00206
  40. CoGS: Controllable Gaussian Splatting. arXiv e-prints, Article arXiv:2312.05664 (2023), arXiv:2312.05664 pages. arXiv:2312.05664
  41. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  42. FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting. arXiv e-prints, Article arXiv:2312.00451 (2023), arXiv:2312.00451 pages. arXiv:2312.00451
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now