Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
43 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ThermalNeRF: Thermal Radiance Fields (2407.15337v1)

Published 22 Jul 2024 in cs.CV

Abstract: Thermal imaging has a variety of applications, from agricultural monitoring to building inspection to imaging under poor visibility, such as in low light, fog, and rain. However, reconstructing thermal scenes in 3D presents several challenges due to the comparatively lower resolution and limited features present in long-wave infrared (LWIR) images. To overcome these challenges, we propose a unified framework for scene reconstruction from a set of LWIR and RGB images, using a multispectral radiance field to represent a scene viewed by both visible and infrared cameras, thus leveraging information across both spectra. We calibrate the RGB and infrared cameras with respect to each other, as a preprocessing step using a simple calibration target. We demonstrate our method on real-world sets of RGB and LWIR photographs captured from a handheld thermal camera, showing the effectiveness of our method at scene representation across the visible and infrared spectra. We show that our method is capable of thermal super-resolution, as well as visually removing obstacles to reveal objects that are occluded in either the RGB or thermal channels. Please see https://yvette256.github.io/thermalnerf for video results as well as our code and dataset release.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. S. Fuentes, E. Tongson, and C. Gonzalez Viejo, “Urban green infrastructure monitoring using remote sensing from integrated visible and thermal infrared cameras mounted on a moving vehicle,” Sensors, vol. 21, no. 1, p. 295, 2021.
  2. M. Lega and R. M. Napoli, “Aerial infrared thermography in the surface waters contamination monitoring,” Desalination and water treatment, vol. 23, no. 1-3, pp. 141–151, 2010.
  3. K. Iwasaki, K. Fukushima, Y. Nagasaka, N. Ishiyama, M. Sakai, and A. Nagasaka, “Real-time monitoring and postprocessing of thermal infrared video images for sampling and mapping groundwater discharge,” Water Resources Research, vol. 59, no. 4, p. e2022WR033630, 2023.
  4. P. Pyykönen, P. Peussa, M. Kutila, and K.-W. Fong, “Multi-camera-based smoke detection and traffic pollution analysis system,” in 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP).   IEEE, 2016, pp. 233–238.
  5. P. Rudol and P. Doherty, “Human body detection and geolocalization for uav search and rescue missions using color and thermal imagery,” in 2008 IEEE aerospace conference.   Ieee, 2008, pp. 1–8.
  6. C. D. Rodin, L. N. de Lima, F. A. de Alcantara Andrade, D. B. Haddad, T. A. Johansen, and R. Storvold, “Object classification in thermal images using convolutional neural networks for search and rescue missions with unmanned aerial systems,” in 2018 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2018, pp. 1–8.
  7. J. Goel, M. Nizamoglu, A. Tan, H. Gerrish, K. Cranmer, N. El-Muttardi, D. Barnes, and P. Dziewulski, “A prospective study comparing the flir one with laser doppler imaging in the assessment of burn depth by a tertiary burns unit in the united kingdom,” Scars, Burns & Healing, vol. 6, p. 2059513120974261, 2020.
  8. M. E. Jaspers, M. Carrière, A. Meij-de Vries, J. Klaessens, and P. Van Zuijlen, “The flir one thermal imager for the assessment of burn wounds: Reliability and validity study,” Burns, vol. 43, no. 7, pp. 1516–1523, 2017.
  9. H. Torresan, B. Turgeon, C. Ibarra-Castanedo, P. Hebert, and X. P. Maldague, “Advanced surveillance systems: combining video and thermal imagery for pedestrian detection,” in Thermosense XXVI, vol. 5405.   SPIE, 2004, pp. 506–515.
  10. A. Akula, R. Ghosh, and H. Sardana, “Thermal imaging and its application in defence systems,” in AIP conference proceedings, vol. 1391, no. 1.   American Institute of Physics, 2011, pp. 333–335.
  11. W. K. Wong, P. N. Tan, C. K. Loo, and W. S. Lim, “An effective surveillance system using thermal camera,” in 2009 international conference on signal acquisition and processing.   IEEE, 2009, pp. 13–17.
  12. J. M. Jurado, A. López, L. Pádua, and J. J. Sousa, “Remote sensing image fusion on 3d scenarios: A review of applications for agriculture and forestry,” International Journal of Applied Earth Observation and Geoinformation, vol. 112, p. 102856, 2022.
  13. R. Näsi, E. Honkavaara, M. Blomqvist, P. Lyytikäinen-Saarenmaa, T. Hakala, N. Viljanen, T. Kantola, and M. Holopainen, “Remote sensing of bark beetle damage in urban forests at individual tree level using a novel hyperspectral camera from uav and aircraft,” Urban Forestry & Urban Greening, vol. 30, pp. 72–83, 2018.
  14. G. T. Miyoshi, M. d. S. Arruda, L. P. Osco, J. Marcato Junior, D. N. Gonçalves, N. N. Imai, A. M. G. Tommaselli, E. Honkavaara, and W. N. Gonçalves, “A novel deep learning method to identify single tree species in uav-based hyperspectral images,” Remote Sensing, vol. 12, no. 8, p. 1294, 2020.
  15. S. Lee, H. Moon, Y. Choi, and D. K. Yoon, “Analyzing thermal characteristics of urban streets using a thermal imaging camera: A case study on commercial streets in seoul, korea,” Sustainability, vol. 10, no. 2, p. 519, 2018.
  16. S. Fuentes, E. J. Tongson, R. De Bei, C. Gonzalez Viejo, R. Ristic, S. Tyerman, and K. Wilkinson, “Non-invasive tools to detect smoke contamination in grapevine canopies, berries and wine: A remote sensing and machine learning modeling approach,” Sensors, vol. 19, no. 15, p. 3335, 2019.
  17. M. Carrasco-Benavides, J. Antunez-Quilobrán, A. Baffico-Hernández, C. Ávila-Sánchez, S. Ortega-Farías, S. Espinoza, J. Gajardo, M. Mora, and S. Fuentes, “Performance assessment of thermal infrared cameras of different resolutions to estimate tree water status from two cherry cultivars: An alternative to midday stem water potential and stomatal conductance,” Sensors, vol. 20, no. 12, p. 3596, 2020.
  18. R. Hernández-Clemente, A. Hornero, M. Mottus, J. Peñuelas, V. González-Dugo, J. C. Jiménez, L. Suárez, L. Alonso, and P. J. Zarco-Tejada, “Early diagnosis of vegetation health from high-resolution hyperspectral and thermal imagery: Lessons learned from empirical relationships and radiative transfer modelling,” Current forestry reports, vol. 5, pp. 169–183, 2019.
  19. V. Malhotra and N. Carino, “Crc handbook on nondestructive testing of concrete,” CRC Press Inc., 01 2004.
  20. M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa, “Nerfstudio: A modular framework for neural radiance field development,” in ACM SIGGRAPH 2023 Conference Proceedings, ser. SIGGRAPH ’23, 2023.
  21. J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  22. J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo,” in European Conference on Computer Vision (ECCV), 2016.
  23. Fluke, “What is thermal imaging? how a thermal image is captured.” [Online]. Available: https://www.fluke.com/en-us/learn/blog/thermal-imaging/how-infrared-cameras-work
  24. ——, “What is thermal imaging? thermal cameras and how they work,” Jan 2024. [Online]. Available: https://www.fluke.com/en-us/learn/blog/thermal-imaging/how-infrared-cameras-work
  25. A. Adán, B. Quintana, J. Garcia Aguilar, V. Pérez, and F. J. Castilla, “Towards the use of 3d thermal models in constructions,” Sustainability, vol. 12, no. 20, p. 8521, 2020.
  26. G. Grechi, M. Fiorucci, G. M. Marmoni, and S. Martino, “3d thermal monitoring of jointed rock masses through infrared thermography and photogrammetry,” Remote Sensing, vol. 13, no. 5, p. 957, 2021.
  27. J. Casana, A. Wiewel, A. Cool, A. C. Hill, K. D. Fisher, and E. J. Laugier, “Archaeological aerial thermography in theory and practice,” Advances in Archaeological Practice, vol. 5, no. 4, pp. 310–327, 2017.
  28. C. Brooke, “Thermal imaging for the archaeological investigation of historic buildings,” Remote Sensing, vol. 10, no. 9, p. 1401, 2018.
  29. B. Mesnik, “Thermal versus optical ip cameras.” [Online]. Available: https://kintronics.com/thermal-versus-optical-ip-cameras/
  30. “Comparing sensitivity of thermal imaging camera modules.” [Online]. Available: https://www.flir.com/discover/cores-components/Comparing-Sensitivity-of-Thermal-Imaging-Cameras-Modules/
  31. R. Schmidt, “How patent-pending technology blends thermal and visible light.” [Online]. Available: https://www.fluke.com/en-us/learn/blog/thermal-imaging/how-patent-pending-technology-blends-thermal-and-visible-light
  32. “Flir one pro thermal imaging camera for smartphones — teledyne flir.” [Online]. Available: https://www.flir.com/products/flir-one-pro/?vertical=condition%2Bmonitoring&segment=solutions
  33. FLIR, “Flir one® series thermal imaging cameras for ios® or android™ smartphones.” [Online]. Available: https://www.flir.com/flir-one/
  34. M. Poggi, P. Zama Ramirez, F. Tosi, S. Salti, L. Di Stefano, and S. Mattoccia, “Cross-spectral neural radiance fields,” in International Conference on 3D Vision, 2022, 3DV.
  35. S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays, “Contactdb: Analyzing and predicting grasp contact via thermal imaging,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8709–8719.
  36. T. Maeda, Y. Wang, R. Raskar, and A. Kadambi, “Thermal non-line-of-sight imaging,” in 2019 IEEE International Conference on Computational Photography (ICCP).   IEEE, 2019, pp. 1–11.
  37. C. Collaro and M. Herkommer, “Research, application, and innovation of lidar technology in spatial archeology,” in Encyclopedia of Information Science and Technology, Sixth Edition.   IGI Global, 2025, pp. 1–33.
  38. C. Collaro, C. Enríquez-Muñoz, A. López, C. Enríquez, and J. M. Jurado, “Detection of landscape features with visible and thermal imaging at the castle of puerta arenas,” Archaeological and Anthropological Sciences, vol. 15, no. 10, p. 152, 2023.
  39. D. Robertson and R. Cipolla, “Practical image processing and computer vision,” in chapter Structure from Motion.   John Wiley & Sons Australia, 2009.
  40. Y. Furukawa, C. Hernández et al., “Multi-view stereo: A tutorial,” Foundations and Trends® in Computer Graphics and Vision, vol. 9, no. 1-2, pp. 1–148, 2015.
  41. S. Wang, H. Jiang, and L. Xiang, “Ct-mvsnet: Efficient multi-view stereo with cross-scale transformer,” in International Conference on Multimedia Modeling.   Springer, 2024, pp. 394–408.
  42. Z. Cheng, C. Esteves, V. Jampani, A. Kar, S. Maji, and A. Makadia, “Lu-nerf: Scene and pose estimation by synchronizing local unposed nerfs,” arXiv preprint arXiv:2306.05410, 2023.
  43. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  44. V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in neural information processing systems, vol. 33, pp. 7462–7473, 2020.
  45. V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, “Deepvoxels: Learning persistent 3d feature embeddings,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2437–2446.
  46. S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh, “Neural volumes: Learning dynamic renderable volumes from images,” arXiv preprint arXiv:1906.07751, 2019.
  47. Sara Fridovich-Keil and Alex Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in CVPR, 2022.
  48. E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis et al., “Efficient geometry-aware 3d generative adversarial networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 123–16 133.
  49. A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” 2022.
  50. T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, no. 4, pp. 102:1–102:15, Jul. 2022. [Online]. Available: https://doi.org/10.1145/3528223.3530127
  51. B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics, vol. 42, no. 4, July 2023. [Online]. Available: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  52. H. Zhu, Y. Sun, C. Liu, L. Xia, J. Luo, N. Qiao, R. Nevatia, and C.-H. Kuo, “Multimodal neural radiance field,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 9393–9399.
  53. Y. Zhang, S. Müller, B. Stephan, H.-M. Gross, and G. Notni, “Point cloud hand–object segmentation using multimodal imaging with thermal and color data for safe robotic object handover,” Sensors, vol. 21, no. 16, p. 5676, 2021.
  54. J. Á. S. Carmona, E. Quirós, V. Mayoral, and C. Charro, “Assessing the potential of multispectral and thermal uav imagery from archaeological sites. a case study from the iron age hillfort of villasviejas del tamuja (cáceres, spain),” Journal of Archaeological Science: Reports, vol. 31, p. 102312, 2020.
  55. M. McLeester, J. Casana, M. R. Schurr, A. C. Hill, and J. H. Wheeler III, “Detecting prehistoric landscape features using thermal, multispectral, and historical imagery analysis at midewin national tallgrass prairie, illinois,” Journal of Archaeological Science: Reports, vol. 21, pp. 450–459, 2018.
  56. G. Patrucco, A. Gómez, A. Adineh, M. Rahrig, and J. L. Lerma, “3d data fusion for historical analyses of heritage buildings using thermal images: The palacio de colomina as a case study,” Remote Sensing, vol. 14, no. 22, p. 5699, 2022.
  57. N. Sutherland, S. Marsh, G. Priestnall, P. Bryan, and J. Mills, “Infrared thermography and 3d-data fusion for architectural heritage: A scoping review,” Remote Sensing, vol. 15, no. 9, p. 2422, 2023.
  58. Microsoft, “Azure kinect dk depth camera.” [Online]. Available: https://learn.microsoft.com/en-us/azure/kinect-dk/depth-camera
  59. J. Oncea, “Swir, mwir, and lwir: One use case for each.” [Online]. Available: https://www.photonicsonline.com/doc/swir-mwir-and-lwir-one-use-case-for-each-0001
  60. I. Electro-Optics, “Nir (near-infrared imaging (fog/haze filter).” [Online]. Available: https://www.infinitioptics.com/technology/nir-near-infrared
  61. O. González, M. I. Lizarraga, S. Karaman, and J. Salas, “Thermal radiation dynamics of soil surfaces with unmanned aerial systems,” in Pattern Recognition: 11th Mexican Conference, MCPR 2019, Querétaro, Mexico, June 26–29, 2019, Proceedings 11.   Springer, 2019, pp. 183–192.
  62. A. López, J. M. Jurado, C. J. Ogayar, and F. R. Feito, “An optimized approach for generating dense thermal point clouds from uav-imagery,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 182, pp. 78–95, 2021.
  63. F. Dümbgen, M. El Helou, N. Gucevska, and S. Süsstrunk, “Near-infrared fusion for photorealistic image dehazing,” Tech. Rep., 2018.
  64. S. Hu, R. Hou, L. Ming, S. Meifang, and P. Chen, “A hyperspectral image reconstruction algorithm based on rgb image using multi-scale atrous residual convolution network,” Frontiers in Marine Science, vol. 9, p. 1006452, 2023.
  65. R. Liu and C. Vondrick, “Humans as light bulbs: 3d human reconstruction from thermal reflection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 12 531–12 542.
  66. N. Max, “Optical models for direct volume rendering,” vol. 1, no. 2, pp. 99–108, 1995.
  67. F. Heide, M. Rouf, M. B. Hullin, B. Labitzke, W. Heidrich, and A. Kolb, “High-quality computational imaging through simple lenses,” ACM Trans. Graph., vol. 32, no. 5, oct 2013. [Online]. Available: https://doi.org/10.1145/2516971.2516974
  68. T. Sun, Y. Peng, and W. Heidrich, “Revisiting cross-channel information transfer for chromatic aberration correction,” in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 3268–3276.
  69. A. Dlesk, K. Vach, and K. Pavelka, “Transformations in the photogrammetric co-processing of thermal infrared images and rgb images,” Sensors, vol. 21, no. 15, p. 5061, 2021.
  70. OpenCV. Camera calibration and 3d reconstruction. [Online]. Available: https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - yvette256/nerfstudio-thermal: Code release for "ThermalNeRF: Thermal Radiance Fields" (7 stars)
  2. ThermalNeRF