Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MixLight: Borrowing the Best of both Spherical Harmonics and Gaussian Models (2404.12768v1)

Published 19 Apr 2024 in cs.CV, cs.AI, and cs.GR

Abstract: Accurately estimating scene lighting is critical for applications such as mixed reality. Existing works estimate illumination by generating illumination maps or regressing illumination parameters. However, the method of generating illumination maps has poor generalization performance and parametric models such as Spherical Harmonic (SH) and Spherical Gaussian (SG) fall short in capturing high-frequency or low-frequency components. This paper presents MixLight, a joint model that utilizes the complementary characteristics of SH and SG to achieve a more complete illumination representation, which uses SH and SG to capture low-frequency ambient and high-frequency light sources respectively. In addition, a special spherical light source sparsemax (SLSparsemax) module that refers to the position and brightness relationship between spherical light sources is designed to improve their sparsity, which is significant but omitted by prior works. Extensive experiments demonstrate that MixLight surpasses state-of-the-art (SOTA) methods on multiple metrics. In addition, experiments on Web Dataset also show that MixLight as a parametric method has better generalization performance than non-parametric methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. P. Debevec, “Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography,” in Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998, pp. 189–198.
  2. G. Li, C. Wu, C. Stoll, Y. Liu, K. Varanasi, Q. Dai, and C. Theobalt, “Capturing relightable human performances under general uncontrolled illumination,” Computer Graphics Forum, vol. 32, no. 2pt3, pp. 275–284, 2013.
  3. S. B. Knorr and D. Kurz, “Real-time illumination estimation from faces for coherent rendering,” in 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).   IEEE, 2014, pp. 113–122.
  4. S. Georgoulis, K. Rematas, T. Ritschel, M. Fritz, T. Tuytelaars, and L. Van Gool, “What is around the camera?” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5170–5178.
  5. J. T. Barron and J. Malik, “Intrinsic scene properties from a single rgb-d image,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 17–24.
  6. L. Gruber, T. Richter-Trummer, and D. Schmalstieg, “Real-time photometric registration from arbitrary geometry,” in 2012 IEEE international symposium on mixed and augmented reality (ISMAR).   IEEE, 2012, pp. 119–128.
  7. L. Bin, X. Kun, and R. R. Martin, “Static scene illumination estimation from videos with applications [j],” Journal of Computer Science and Technology, vol. 32, no. 3, pp. 430–442, 2017.
  8. M.-A. Gardner, K. Sunkavalli, E. Yumer, X. Shen, E. Gambaretto, C. Gagné, and J.-F. Lalonde, “Learning to predict indoor illumination from a single image,” ACM Transactions on Graphics, vol. 36, no. 6, pp. 1–14, 2017.
  9. G. Wang, Y. Yang, C. C. Loy, and Z. Liu, “Stylelight: Hdr panorama generation for lighting estimation and editing,” in European Conference on Computer Vision, 2022, pp. 477–492.
  10. F. Zhan, C. Zhang, Y. Yu, Y. Chang, S. Lu, F. Ma, and X. Xie, “Emlight: Lighting estimation via spherical distribution approximation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 4, 2021, pp. 3287–3295.
  11. F. Zhan, Y. Yu, C. Zhang, R. Wu, W. Hu, S. Lu, F. Ma, X. Xie, and L. Shao, “Gmlight: Lighting estimation via geometric distribution approximation,” IEEE Transactions on Image Processing, vol. 31, pp. 2268–2278, 2022.
  12. M. R. K. Dastjerdi, J. Eisenmann, Y. Hold-Geoffroy, and J.-F. Lalonde, “Everlight: Indoor-outdoor editable hdr lighting estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7420–7429.
  13. H. Weber, M. Garon, and J.-F. Lalonde, “Editable indoor lighting estimation,” in European Conference on Computer Vision.   Springer, 2022, pp. 677–692.
  14. M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J.-F. Lalonde, “Deep parametric indoor lighting estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 7175–7183.
  15. M. Li, J. Guo, X. Cui, R. Pan, Y. Guo, C. Wang, P. Yu, and F. Pan, “Deep spherical gaussian illumination estimation for indoor scene,” in Proceedings of the ACM Multimedia Asia, 2019, pp. 1–6.
  16. F. Zhan, C. Zhang, W. Hu, S. Lu, F. Ma, X. Xie, and L. Shao, “Sparse needlets for lighting estimation with spherical transport loss,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 830–12 839.
  17. M. Garon, K. Sunkavalli, S. Hadap, N. Carr, and J.-F. Lalonde, “Fast spatially-varying indoor lighting estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 6908–6917.
  18. D. Xu, Z. Li, and Y. Zhang, “Real-time illumination estimation for mixed reality on mobile devices,” in 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).   IEEE, 2020, pp. 702–703.
  19. D. Cheng, J. Shi, Y. Chen, X. Deng, and X. Zhang, “Learning scene illumination by pairwise photos from rear and front mobile cameras,” in Computer Graphics Forum, vol. 37, no. 7.   Wiley Online Library, 2018, pp. 213–221.
  20. A. Martins and R. Astudillo, “From softmax to sparsemax: A sparse model of attention and multi-label classification,” in International conference on machine learning.   PMLR, 2016, pp. 1614–1623.
  21. A. Laha, S. A. Chemmengath, P. Agrawal, M. Khapra, K. Sankaranarayanan, and H. G. Ramaswamy, “On controllable sparse alternatives to softmax,” Advances in neural information processing systems, vol. 31, 2018.
  22. Y. Mei, H. Zhang, X. Zhang, J. Zhang, Z. Shu, Y. Wang, Z. Wei, S. Yan, H. Jung, and V. M. Patel, “Lightpainter: interactive portrait relighting with freehand scribble,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 195–205.
  23. T. Nestmeyer, J.-F. Lalonde, I. Matthews, and A. Lehrmann, “Learning physics-guided face relighting under directional light,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5124–5133.
  24. R. Pandey, S. Orts-Escolano, C. Legendre, C. Haene, S. Bouaziz, C. Rhemann, P. E. Debevec, and S. R. Fanello, “Total relighting: learning to relight portraits for background replacement.” ACM Trans. Graph., vol. 40, no. 4, pp. 43–1, 2021.
  25. T. Sun, J. T. Barron, Y.-T. Tsai, Z. Xu, X. Yu, G. Fyffe, C. Rhemann, J. Busch, P. Debevec, and R. Ramamoorthi, “Single image portrait relighting,” ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–12, 2019.
  26. Z. Wang, X. Yu, M. Lu, Q. Wang, C. Qian, and F. Xu, “Single image portrait relighting via explicit multiple reflectance channel modeling,” ACM Transactions on Graphics (TOG), vol. 39, no. 6, pp. 1–13, 2020.
  27. Y.-Y. Yeh, K. Nagano, S. Khamis, J. Kautz, M.-Y. Liu, and T.-C. Wang, “Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation,” ACM Transactions on Graphics (TOG), vol. 41, no. 6, pp. 1–21, 2022.
  28. L. Zhang, Q. Zhang, M. Wu, J. Yu, and L. Xu, “Neural video portrait relighting in real-time via consistency modeling,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 802–812.
  29. X. Zhang, S. Fanello, Y.-T. Tsai, T. Sun, T. Xue, R. Pandey, S. Orts-Escolano, P. Davidson, C. Rhemann, P. Debevec et al., “Neural light transport for relighting and view synthesis,” ACM Transactions on Graphics (TOG), vol. 40, no. 1, pp. 1–17, 2021.
  30. A. Meka, C. Haene, R. Pandey, M. Zollhöfer, S. Fanello, G. Fyffe, A. Kowdle, X. Yu, J. Busch, J. Dourgarian et al., “Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination,” ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–12, 2019.
  31. H. Kim, M. Jang, W. Yoon, J. Lee, D. Na, and S. Woo, “Switchlight: Co-design of physics-driven architecture and pre-training framework for human portrait relighting,” arXiv preprint arXiv:2402.18848, 2024.
  32. H. Weber, D. Prévost, and J.-F. Lalonde, “Learning to estimate indoor lighting from 3d objects,” in 2018 International Conference on 3D Vision (3DV).   IEEE, 2018, pp. 199–207.
  33. H.-X. Yu, S. Agarwala, C. Herrmann, R. Szeliski, N. Snavely, J. Wu, and D. Sun, “Accidental light probes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 12 521–12 530.
  34. T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2337–2346.
  35. T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8110–8119.
  36. H. Zhu, H. Yang, L. Guo, Y. Zhang, Y. Wang, M. Huang, M. Wu, Q. Shen, R. Yang, and X. Cao, “Facescape: 3d facial dataset and benchmark for single-view 3d face reconstruction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  37. Y. Feng, H. Feng, M. J. Black, and T. Bolkart, “Learning an animatable detailed 3d face model from in-the-wild images,” ACM Transactions on Graphics (ToG), vol. 40, no. 4, pp. 1–13, 2021.
  38. Z. Chen, A. Chen, G. Zhang, C. Wang, Y. Ji, K. N. Kutulakos, and J. Yu, “A neural rendering framework for free-viewpoint relighting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5599–5610.
  39. H. Zhou, S. Hadap, K. Sunkavalli, and D. W. Jacobs, “Deep single-image portrait relighting,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7194–7202.
  40. D. Frolova, D. Simakov, and R. Basri, “Accuracy of spherical harmonic approximations for images of lambertian objects under far and near lighting,” in Computer Vision-ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part I 8.   Springer, 2004, pp. 574–587.
  41. R. Ramamoorthi, “Analytic pca construction for theoretical analysis of lighting variability in images of a lambertian object,” IEEE transactions on pattern analysis and machine intelligence, vol. 24, no. 10, pp. 1322–1333, 2002.
  42. ——, “Modeling illumination variation with spherical harmonics,” Face Processing: Advanced Modeling Methods, pp. 385–424, 2006.
  43. H. Vogel, “A better way to construct the sunflower head,” Mathematical biosciences, vol. 44, no. 3-4, pp. 179–189, 1979.
  44. Q. Zhao, P. Tan, Q. Dai, L. Shen, E. Wu, and S. Lin, “A closed-form solution to retinex with nonlocal texture constraints,” IEEE transactions on pattern analysis and machine intelligence, vol. 34, no. 7, pp. 1437–1444, 2012.
  45. R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman, “Ground truth dataset and baseline evaluations for intrinsic image algorithms,” in 2009 IEEE 12th International Conference on Computer Vision, 2009, pp. 2335–2342.
  46. S. Sengupta, A. Kanazawa, C. D. Castillo, and D. W. Jacobs, “Sfsnet: Learning shape, reflectance and illuminance of facesin the wild’,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6296–6305.
  47. A. Meka, M. Shafiei, M. Zollhöfer, C. Richardt, and C. Theobalt, “Real-time global illumination decomposition of videos,” ACM Transactions on Graphics (ToG), vol. 40, no. 3, pp. 1–16, 2021.
  48. R. Wang, Q. Zhang, C.-W. Fu, X. Shen, W.-S. Zheng, and J. Jia, “Underexposed photo enhancement using deep illumination estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6849–6857.
  49. V. Gkitsas, N. Zioulis, F. Alvarez, D. Zarpalas, and P. Daras, “Deep lighting environment map estimation from spherical panoramas,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 640–641.
  50. R. Ramamoorthi and P. Hanrahan, “An efficient representation for irradiance environment maps,” in Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001, pp. 497–500.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets