Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering (2401.06003v2)

Published 11 Jan 2024 in cs.CV and cs.GR

Abstract: Point-based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al. 2023] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen-space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole-free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions. Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art methods in terms of rendering quality while maintaining a real-time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto-exposed footage. The project page is located at: https://lfranke.github.io/trips/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Point-based computer graphics. In ACM SIGGRAPH 2004 Course Notes. 2004, pp. 7–es.
  2. Neural point-based graphics. In European Conference on Computer Vision (2020), Springer, pp. 696–712.
  3. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2021), pp. 5855–5864.
  4. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 5470–5479.
  5. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 (2023).
  6. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 7911–7920.
  7. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics (TOG) 32, 3 (2013), 1–12.
  8. Tensorf: Tensorial radiance fields. In Computer Vision – ECCV 2022 (Cham, 2022), Avidan S., Brostow G., Cissé M., Farinella G. M., Hassner T., (Eds.), Springer Nature Switzerland, pp. 333–350.
  9. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 14124–14133.
  10. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1.
  11. Efficient view-dependent ibr with projective texture-mapping. In EG Rendering Workshop (1998), vol. 4.
  12. Multi-layer depth of field rendering with tiled splatting. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 1–17.
  13. Plenoxels: Radiance fields without neural networks. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), pp. 5491–5500. doi:10.1109/CVPR52688.2022.00542.
  14. Deepstereo: Learning to predict new views from the world’s imagery. In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 5515–5524.
  15. Grossman J. P., Dally W. J.: Point sample rendering. In Eurographics Workshop on Rendering Techniques (1998), Springer, pp. 181–192.
  16. Deepwarp: Photorealistic image resynthesis for gaze manipulation. In European conference on computer vision (2016), Springer, pp. 311–326.
  17. Multi-view stereo for community photo collections. In 2007 IEEE 11th International Conference on Computer Vision (2007), IEEE, pp. 1–8.
  18. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–15.
  19. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 5875–5884.
  20. Perceptual losses for real-time style transfer and super-resolution. CoRR abs/1603.08155 (2016). URL: http://arxiv.org/abs/1603.08155.
  21. Kobbelt L., Botsch M.: A survey of point-based techniques in computer graphics. Computers & Graphics 28, 6 (2004), 801–814.
  22. Kopanas G., Drettakis G.: Improving NeRF Quality by Progressive Camera Placement for Free-Viewpoint Navigation. In Vision, Modeling, and Visualization (2023), Guthe M., Grosch T., (Eds.), The Eurographics Association. doi:10.2312/vmv.20231222.
  23. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42, 4 (2023).
  24. Real-time 3D reconstruction in dynamic scenes using point-based fusion. In Proc. of Joint 3DIM/3DPVT Conference (3DV) (June 2013), pp. 1–8. Selected for oral presentation.
  25. Neural point catacaustics for novel-view synthesis of reflections. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–15.
  26. Point-based neural rendering with per-view optimization. In Computer Graphics Forum (2021), vol. 40, Wiley Online Library, pp. 29–43.
  27. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics 36, 4 (2017).
  28. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 3 (2022), 3292–3310.
  29. Lassner C., Zollhöfer M.: Pulsar: Efficient sphere-based neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2021), pp. 1440–1449.
  30. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 7210–7219.
  31. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989 (2022).
  32. Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 6878–6887.
  33. Efficient point-based rendering using image reconstruction. In PBG@ Eurographics (2007), pp. 101–108.
  34. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14.
  35. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65, 1 (2021), 99–106.
  36. Müller T.: tiny-cuda-nn, 4 2021. URL: https://github.com/NVlabs/tiny-cuda-nn.
  37. Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks. In Computer Graphics Forum (2021), vol. 40, Wiley Online Library, pp. 45–59.
  38. Real-time rendering of massive unstructured raw point clouds using screen-space operators. In Proceedings of the 12th International conference on Virtual Reality, Archaeology and Cultural Heritage (2011), pp. 105–112.
  39. Penner E., Zhang L.: Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–11.
  40. Surfels: Surface elements as rendering primitives. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (2000), pp. 335–342.
  41. NPBG++: accelerating neural point-based graphics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (June 2022), pp. 15969–15979.
  42. Adop: Approximate differentiable one-pixel point rendering. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–14.
  43. Riegler G., Koltun V.: Free view synthesis. In European Conference on Computer Vision (2020), Springer, pp. 623–640.
  44. Riegler G., Koltun V.: Stable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 12216–12225.
  45. Neat: Neural adaptive tomography. ACM Trans. Graph. 41, 4 (jul 2022). URL: https://doi.org/10.1145/3528223.3530121, doi:10.1145/3528223.3530121.
  46. Deep novel view synthesis from colored 3d point clouds. In European Conference on Computer Vision (2020), Springer, pp. 1–17.
  47. Schonberger J. L., Frahm J.-M.: Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 4104–4113.
  48. Shum H., Kang S. B.: Review of image-based rendering techniques. In Visual Communications and Image Processing 2000 (2000), vol. 4067, SPIE, pp. 2–13.
  49. Real-time continuous level of detail rendering of point clouds. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (2019), IEEE, pp. 103–110.
  50. Rendering point clouds with compute shaders and vertex order optimization. In Computer Graphics Forum (2021), vol. 40, Wiley Online Library, pp. 115–126.
  51. Software rasterization of 2 billion points in real time. arXiv preprint arXiv:2204.01287 (2022).
  52. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems 33 (2020).
  53. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers. 2006, pp. 835–846.
  54. Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 175–184.
  55. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), pp. 2437–2446.
  56. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV) (2016).
  57. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 2846–2855.
  58. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 12922–12931.
  59. Tucker R., Snavely N.: Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 551–560.
  60. Advances in neural rendering. In Computer Graphics Forum (2022), vol. 41, Wiley Online Library, pp. 703–735.
  61. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.
  62. A survey of multifragment rendering. In Computer Graphics Forum (2020), vol. 39, Wiley Online Library, pp. 623–642.
  63. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 7467–7477.
  64. Elasticfusion: Real-time dense slam and light source estimation. The International Journal of Robotics Research 35, 14 (2016), 1697–1716.
  65. Surfelgan: Synthesizing realistic sensor data for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020).
  66. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 5752–5761.
  67. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision (2019), pp. 4471–4480.
  68. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–14.
  69. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 4578–4587.
  70. Differentiable point-based radiance fields for efficient view synthesis. arXiv preprint arXiv:2205.14330 (2022).
  71. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR (2018).
  72. Surface splatting. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001), pp. 371–378.
  73. Stereo magnification: Learning view synthesis using multiplane images. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–12.
  74. View synthesis by appearance flow. In European conference on computer vision (2016), Springer, pp. 286–301.
Citations (15)
View on Semantic Scholar

Summary

  • The paper introduces a novel trilinear point splatting approach that renders detailed radiance fields in real-time.
  • It combines an image pyramid rasterization with a lightweight neural network to optimize point sizes and eliminate artifacts.
  • TRIPS delivers consistent 60 fps performance and temporal consistency, making it ideal for gaming, VR, and interactive media.

Introduction

Point-based rendering techniques have revolutionized the way 3D scenes are visualized, offering a combination of high-quality images and computational efficiency. These methods are adept at handling a range of scenes but often face challenges like artifact mitigation and maintaining real-time performance. TRIPS—Trilinear Point Splatting—is introduced as an innovative approach combining the strengths of existing methods to render detailed radiance fields in real-time while filling in substantial gaps and ensuring temporal consistency.

The TRIPS Method

TRIPS stands out by rasterizing points into an image pyramid mapped to screen space, which adapts to the rendered point size, allowing for rendering large points efficiently with a single trilinear write. Following this, a lightweight neural network reconstructs a detailed, hole-free image. What sets TRIPS apart is the wholly differentiable pipeline, which enables automatic optimization of point sizes and positions to enhance scene representation precision.

Performance and Applications

The evaluation of TRIPS has shown that it excels at producing images of superior quality at a consistent real-time frame rate of 60 fps on standard hardware configurations. TRIPS is not just about speed; it also handles rendering in scenarios with complex geometries and even processes footages with variable exposures effectively. The algorithm's versatility is further supported by an entirely differentiable pipeline, leading to greater optimization possibilities.

Future Outlook and Sources

TRIPS establishes a new benchmark in terms of visual fidelity and performance for real-time radiance field rendering. By adeptly balancing quality rendering with real-time capabilities, it has potential applications in various industries, including gaming, virtual reality, and interactive media. For those interested in exploring the technical depths or implementing the method, resources and a full implementation are made accessible at the project repository: TRIPS on GitHub.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - lfranke/TRIPS (524 stars)
  2. TRIPS: Trilinear Point Splatting
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube