Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neural Rendering and Its Hardware Acceleration: A Review (2402.00028v1)

Published 6 Jan 2024 in cs.GR, cs.CV, and eess.IV

Abstract: Neural rendering is a new image and video generation method based on deep learning. It combines the deep learning model with the physical knowledge of computer graphics, to obtain a controllable and realistic scene model, and realize the control of scene attributes such as lighting, camera parameters, posture and so on. On the one hand, neural rendering can not only make full use of the advantages of deep learning to accelerate the traditional forward rendering process, but also provide new solutions for specific tasks such as inverse rendering and 3D reconstruction. On the other hand, the design of innovative hardware structures that adapt to the neural rendering pipeline breaks through the parallel computing and power consumption bottleneck of existing graphics processors, which is expected to provide important support for future key areas such as virtual and augmented reality, film and television creation and digital entertainment, artificial intelligence and the metaverse. In this paper, we review the technical connotation, main challenges, and research progress of neural rendering. On this basis, we analyze the common requirements of neural rendering pipeline for hardware acceleration and the characteristics of the current hardware acceleration architecture, and then discuss the design challenges of neural rendering processor architecture. Finally, the future development trend of neural rendering processor architecture is prospected.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (135)
  1. Neural point-based graphics. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pages 696–712. Springer, 2020.
  2. AMD. Amd compare graphics specifications. 2023.
  3. Hypergraph propagation and community selection for objects retrieval. Advances in Neural Information Processing Systems, 34, 2021.
  4. Towards content-based pixel retrieval in revisited oxford and paris. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20507–20518, 2023.
  5. Topological ransac for instance verification and retrieval without fine-tuning. arXiv preprint arXiv:2310.06486, 2023.
  6. Kernel-predicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph., 36(4):97–1, 2017.
  7. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20919–20929, 2023.
  8. Deep cg2real: Synthetic-to-real translation via image disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2730–2739, 2019.
  9. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824, 2020.
  10. Cambricon. Cambricon mlu370 chip. 2023.
  11. Real-time neural light field on mobile devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8328–8337, 2023.
  12. Immfusion: Robust mmwave-rgb fusion for 3d human body reconstruction in all weather conditions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2752–2758. IEEE, 2023.
  13. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16569–16578, 2023.
  14. Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  15. Weakly-supervised contrastive learning in path manifold for monte carlo image reconstruction. ACM Transactions on Graphics (TOG), 40(4):38–1, 2021.
  16. J. Choquette. Nvidia hopper gpu: Scaling performance. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–46. IEEE Computer Society, 2022.
  17. B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303–312, 1996.
  18. Scalable realistic rendering with many-light methods. In Computer Graphics Forum, volume 33, pages 88–104. Wiley Online Library, 2014.
  19. Neural point cloud rendering via multi-plane projection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7830–7839, 2020.
  20. Neural scene representation and rendering. Science, 360(6394):1204–1210, 2018.
  21. Real-time monte carlo denoising with weight sharing kernel prediction network. In Computer Graphics Forum, volume 40, pages 15–27. Wiley Online Library, 2021.
  22. Gen-nerf: Efficient and generalizable neural radiance fields via algorithm-hardware co-design. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–12, 2023.
  23. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346–14355, 2021.
  24. The irradiance volume. IEEE Computer Graphics and Applications, 18(2):32–43, 1998.
  25. Neural radiosity. ACM Transactions on Graphics (TOG), 40(6):1–11, 2021.
  26. An interactive image-based modeling system. arXiv preprint arXiv:2203.14441, 2022.
  27. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875–5884, 2021.
  28. A new golden age for computer architecture. Communications of the ACM, 62(2):48–60, 2019.
  29. M. Hong and L. Xu. Biren br100 gpgpu: Accelerating datacenter scale ai computing. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–22. IEEE Computer Society, 2022.
  30. Y. Huo. Extension–adaptive sampling with implicit radiance field. arXiv preprint arXiv:2202.00855, 2022.
  31. Spherical gaussian-based lightcuts for glossy interreflections. In Computer Graphics Forum, volume 39, pages 192–203. Wiley Online Library, 2020.
  32. Shadertransformer: Predicting shader quality via one-shot embedding for fast simplification. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  33. Adaptive matrix column sampling and completion for rendering participating media. ACM Transactions on Graphics (TOG), 35(6):1–11, 2016.
  34. A matrix sampling-and-recovery approach for many-lights rendering. ACM Transactions on Graphics (TOG), 34(6):1–12, 2015.
  35. Sparse sampling and completion for light transport in vpl-based rendering. arXiv preprint arXiv:2202.12567, 2022.
  36. Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning. ACM Transactions on Graphics (TOG), 39(1):1–17, 2020.
  37. Y. Huo and S.-e. Yoon. A survey on deep learning-based monte carlo denoising. Computational Visual Media, 7(2):169–185, 2021.
  38. H. Jiang. Intel’s ponte vecchio gpu: Architecture, systems & software. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–29. IEEE Computer Society, 2022.
  39. A virtual point light generation method in close-range area. arXiv preprint arXiv:2203.11484, 2022.
  40. A survey on visual language pre-training. Journal of Software, 34(5):0–0, 2022.
  41. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18365–18375, 2022.
  42. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–14, 2023.
  43. Ten lessons from three generations shaped google’s tpuv4i: Industrial product. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 1–14. IEEE, 2021.
  44. J. T. Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pages 143–150, 1986.
  45. R. Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing, 29(6):1153–1160, 1981.
  46. Single image reflection removal with physically-based training images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5164–5173, 2020.
  47. Point-based neural rendering with per-view optimization. In Computer Graphics Forum, volume 40, pages 29–43. Wiley Online Library, 2021.
  48. Dp-nerf: Deblurred neural radiance field with physical scene priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12386–12396, 2023.
  49. Contact2grasp: 3d grasp synthesis via hand-object contact constraint. arXiv preprint, 2022.
  50. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–13, 2023.
  51. Automatic band-limited approximation of shaders using mean-variance statistics in clamped domain. In Computer Graphics Forum, volume 39, pages 181–192. Wiley Online Library, 2020.
  52. Multi-resolution terrain rendering using summed-area tables. Computers & Graphics, 95:130–140, 2021.
  53. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6508, 2021.
  54. Automatic mesh and shader level of detail. IEEE Transactions on Visualization and Computer Graphics, 2022.
  55. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14556–14565, 2021.
  56. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651–15663, 2020.
  57. Conical emission induced by the filamentation of femtosecond vortex beams in water. Applied Sciences, 13(22):12435, 2023.
  58. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751, 2019.
  59. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  60. Image reconstruction by convolution with symmetrical piecewise nth-order polynomial kernels. IEEE transactions on image processing, 8(2):192–201, 1999.
  61. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  62. Hardware acceleration of neural graphics. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–12, 2023.
  63. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  64. Neural control variates. ACM Transactions on Graphics (TOG), 39(6):1–19, 2020.
  65. Deep shading: convolutional neural networks for screen space shading. In Computer graphics forum, volume 36, pages 65–78. Wiley Online Library, 2017.
  66. Rendernet: A deep convolutional network for differentiable rendering from 3d shapes. Advances in neural information processing systems, 31, 2018.
  67. M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  68. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504–3515, 2020.
  69. NVIDIA. Nvidia ada gpu architecture. 2023.
  70. NVIDIA. Nvidia dlss 3. 2023.
  71. NVIDIA. Rtx technology. 2023.
  72. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531–4540, 2019.
  73. 3.3 kunlun: A 14nm high-performance ai processor for diversified workloads. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), volume 64, pages 50–51. IEEE, 2021.
  74. Meshchain: Secure 3d model and intellectual property management powered by blockchain technology. In Computer Graphics International Conference, pages 519–534. Springer, 2021.
  75. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  76. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  77. Icarus: A specialized architecture for neural radiance fields rendering. ACM Transactions on Graphics (TOG), 41(6):1–14, 2022.
  78. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335–14345, 2021.
  79. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901–10911, 2021.
  80. K. Rematas and V. Ferrari. Neural voxel renderer: Learning an accurate and controllable rendering tool. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5417–5427, 2020.
  81. Data-driven digital lighting design for residential indoor spaces. ACM Transactions on Graphics, 42(3):1–18, 2023.
  82. Minervas: Massive interior environments virtual synthesis. In Computer Graphics Forum, volume 41, pages 63–74. Wiley Online Library, 2022.
  83. Supplementary material for minervas: Massive interior environments virtual synthesis. 2022.
  84. L. Ren and Y. Song. Aogan: A generative adversarial network for screen space ambient occlusion. Computational Visual Media, 8(3):483–494, 2022.
  85. Ai and ml accelerator survey and trends. In 2022 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–10. IEEE, 2022.
  86. Learning neural light transport. arXiv preprint arXiv:2006.03427, 2020.
  87. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313–19325, 2021.
  88. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  89. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
  90. A. Skillman and T. Edso. A technical overview of cortex-m55 and ethos-u55: Arm’s most capable processors for endpoint ai. In 2020 IEEE Hot Chips 32 Symposium (HCS), pages 1–20. IEEE Computer Society, 2020.
  91. A. Smith and N. James. Amd instinct™ mi200 series accelerator and node architectures. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–23. IEEE Computer Society, 2022.
  92. Nelf: Neural light-transport field for portrait view synthesis and relighting. arXiv preprint arXiv:2107.12351, 2021.
  93. Neural screen space rendering of direct illumination. Pacific Graphics Short Papers, Posters, and Work-in-Progress Papers, 2021.
  94. Dojo: The microarchitecture of tesla’s exa-scale computer. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–28. IEEE Computer Society, 2022.
  95. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248–8258, 2022.
  96. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pages 701–727. Wiley Online Library, 2020.
  97. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703–735. Wiley Online Library, 2022.
  98. Deep illumination: Approximating dynamic global illumination with generative adversarial network. arXiv preprint arXiv:1710.09834, 2017.
  99. Energy efficiency boost in the ai-infused power10 processor. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 29–42. IEEE, 2021.
  100. W. Trina. Truly global illumination: ray tracing for the masses. 2023.
  101. R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis. In European Conference on Computer Vision, pages 612–629. Springer, 2022.
  102. A biophysically-based skin model for heterogeneous volume rendering.
  103. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  104. State of the art on deep learning-enhanced rendering methods. Machine Intelligence Research, 20(6):799–821, 2023.
  105. Real-time rendering and editing of scattering effects for translucent objects. arXiv preprint arXiv:2203.12339, 2022.
  106. Variational hierarchical directed bounding box construction for solid mesh models. arXiv preprint arXiv:2203.10521, 2022.
  107. Implementation details of gpu-based out-of-core many-lights rendering.
  108. Gpu-based out-of-core many-lights rendering. ACM Transactions on Graphics (TOG), 32(6):1–10, 2013.
  109. Automatic shader simplification using surface signal approximation. ACM Transactions on Graphics (TOG), 33(6):1–11, 2014.
  110. Real-time rendering on a power budget. ACM Transactions on Graphics (TOG), 35(4):1–11, 2016.
  111. Seal-3d: Interactive pixel-level editing for neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17683–17693, 2023.
  112. G. Z. Wang Endong, Yan Ruidong. A survey of distributed training system and its optimization algorithms. 2023.
  113. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7467–7477, 2020.
  114. Adaptive recurrent frame prediction with learnable motion vectors. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023.
  115. Holistic inverse rendering of complex facade via aerial 3d scanning. arXiv preprint arXiv:2311.11825, 2023.
  116. Adversarial monte carlo denoising with conditioned auxiliary feature modulation. ACM Trans. Graph., 38(6):224–1, 2019.
  117. Rule-based procedural tree modeling approach. arXiv preprint arXiv:2204.03237, 2022.
  118. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  119. Contour-aware equipotential earning for semantic segmentation. IEEE Transactions on Multimedia, 2022.
  120. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752–5761, 2021.
  121. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  122. Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5682–5692, 2023.
  123. Comprehensive review of visual-language-oriented multimodal pretraining methods. Journal of Image and Graphics, 27(9):2652–2682, 2022.
  124. Neural light transport for relighting and view synthesis. ACM Transactions on Graphics (TOG), 40(1):1–17, 2021.
  125. Powernet: Learning-based real-time power-budget rendering. IEEE Transactions on Visualization and Computer Graphics, 2021.
  126. Nelt: Object-oriented neural light transfer. ACM Transactions on Graphics, 2023.
  127. Normal and visibility estimation of human face from a single image. arXiv preprint arXiv:2203.04647, 2022.
  128. Morphological anti-aliasing method for boundary slope prediction. arXiv preprint arXiv:2203.03870, 2022.
  129. Neural super-resolution in real-time rendering using auxiliary feature enhancement. Journal of Database Management (JDM), 34(3):1–13, 2023.
  130. Fusesr: Super resolution for real-time rendering through efficient multi-resolution fusion. In SIGGRAPH Asia 2023 Conference Papers, pages 1–10, 2023.
  131. Review of convolutional neural network. 2017.
  132. Supplementary material: I2-sdf: Intrinsic indoor scene reconstruction and editing via raytracing in neural sdfs.
  133. I2-sdf: Intrinsic indoor scene reconstruction and editing via raytracing in neural sdfs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12489–12498, 2023.
  134. Learning-based inverse rendering of complex indoor scenes with differentiable monte carlo raytracing. In SIGGRAPH Asia 2022 Conference Papers, pages 1–8, 2022.
  135. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now