Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unsupervised HDR Image and Video Tone Mapping via Contrastive Learning (2303.07327v2)

Published 13 Mar 2023 in cs.CV and eess.IV

Abstract: Capturing high dynamic range (HDR) images (videos) is attractive because it can reveal the details in both dark and bright regions. Since the mainstream screens only support low dynamic range (LDR) content, tone mapping algorithm is required to compress the dynamic range of HDR images (videos). Although image tone mapping has been widely explored, video tone mapping is lagging behind, especially for the deep-learning-based methods, due to the lack of HDR-LDR video pairs. In this work, we propose a unified framework (IVTMNet) for unsupervised image and video tone mapping. To improve unsupervised training, we propose domain and instance based contrastive learning loss. Instead of using a universal feature extractor, such as VGG to extract the features for similarity measurement, we propose a novel latent code, which is an aggregation of the brightness and contrast of extracted features, to measure the similarity of different pairs. We totally construct two negative pairs and three positive pairs to constrain the latent codes of tone mapped results. For the network structure, we propose a spatial-feature-enhanced (SFE) module to enable information exchange and transformation of nonlocal regions. For video tone mapping, we propose a temporal-feature-replaced (TFR) module to efficiently utilize the temporal correlation and improve the temporal consistency of video tone-mapped results. We construct a large-scale unpaired HDR-LDR video dataset to facilitate the unsupervised training process for video tone mapping. Experimental results demonstrate that our method outperforms state-of-the-art image and video tone mapping methods. Our code and dataset are available at https://github.com/cao-cong/UnCLTMO.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Y. Ou, P. Ambalathankandy, M. Ikebe, S. Takamaeda, M. Motomura, and T. Asai, “Real-time tone mapping: A state of the art report,” arXiv preprint arXiv:2003.03074, 2020.
  2. Y. Ou, P. Ambalathankandy, S. Takamaeda, M. Motomura, T. Asai, and M. Ikebe, “Real-time tone mapping: a survey and cross-implementation hardware benchmark,” IEEE Transactions on Circuits and Systems for Video Technology, 2021.
  3. P. Ambalathankandy, M. Ikebe, T. Yoshida, T. Shimada, S. Takamaeda, M. Motomura, and T. Asai, “An adaptive global and local tone mapping algorithm implemented on fpga,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 9, pp. 3015–3028, 2019.
  4. C. Jung and T. Sun, “Optimized perceptual tone mapping for contrast enhancement of images,” IEEE Transactions on circuits and systems for video technology, vol. 27, no. 6, pp. 1161–1170, 2016.
  5. K. S. Song and M. G. Kang, “Optimized tone mapping function for contrast enhancement considering human visual perception system,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 11, pp. 3199–3210, 2018.
  6. G. Ward, “A contrast-based scalefactor for luminance display.” Graphics Gems, vol. 4, pp. 415–21, 1994.
  7. J. Tumblin and H. Rushmeier, “Tone reproduction for realistic images,” IEEE Computer graphics and Applications, vol. 13, no. 6, pp. 42–48, 1993.
  8. F. Drago, K. Myszkowski, T. Annen, and N. Chiba, “Adaptive logarithmic mapping for displaying high contrast scenes,” in Computer graphics forum, vol. 22, no. 3.   Wiley Online Library, 2003, pp. 419–426.
  9. C. Schlick, “Quantization techniques for visualization of high dynamic range pictures,” in Photorealistic rendering techniques.   Springer, 1995, pp. 7–20.
  10. Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski, “Edge-preserving decompositions for multi-scale tone and detail manipulation,” ACM transactions on graphics (TOG), vol. 27, no. 3, pp. 1–10, 2008.
  11. B. Gu, W. Li, M. Zhu, and M. Wang, “Local edge-preserving multiscale decomposition for high dynamic range image tone mapping,” IEEE Transactions on Image Processing, vol. 22, no. 1, pp. 70–79, 2013.
  12. T. Shibata, T. Masayuki, and M. Okutomi, “Gradient-domain image reconstruction framework with intensity-range and base-structure constraints,” 06 2016, pp. 2745–2753.
  13. Z. Liang, J. Xu, D. Zhang, Z. Cao, and L. Zhang, “A hybrid l1-l0 layer decomposition model for tone mapping,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4758–4766.
  14. X. Cao, K. Lai, S. N. Yanushkevich, and M. R. Smith, “Adversarial and adaptive tone mapping operator for high dynamic range images,” in 2020 IEEE Symposium Series on Computational Intelligence (SSCI).   IEEE, 2020, pp. 1814–1821.
  15. K. Panetta, L. Kezebou, V. Oludare, S. Agaian, and Z. Xia, “Tmo-net: A parameter-free tone mapping operator using generative adversarial network, and performance benchmarking on large scale hdr dataset,” IEEE Access, vol. 9, pp. 39 500–39 517, 2021.
  16. Y. Vinker, I. Huberman-Spiegelglas, and R. Fattal, “Unpaired learning for high dynamic range image tone mapping,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 657–14 666.
  17. S. N. Pattanaik, J. Tumblin, H. Yee, and D. P. Greenberg, “Time-dependent visual adaptation for fast realistic image display,” in Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000, pp. 47–54.
  18. R. Boitard, K. Bouatouch, R. Cozot, D. Thoreau, and A. Gruson, “Temporal coherency for video tone mapping,” in Applications of Digital Image Processing XXXV, vol. 8499.   SPIE, 2012, pp. 113–122.
  19. Q. Shan, M. Meyer, T. DeRose, and J. Anderson, “Tone mapping high dynamic range videos using wavelets pixar technical memo 1201,” 2012.
  20. C. Kiser, E. Reinhard, M. Tocci, and N. Tocci, “Real time automated tone mapping system for hdr video,” in IEEE International Conference on Image Processing, vol. 134.   IEEE Orlando, FL, 2012, pp. 2749–2752.
  21. G. Eilertsen, J. Unger, R. Wanat, and R. Mantiuk, “Survey and evaluation of tone mapping operators for hdr video,” in ACM SIGGRAPH 2013 Talks, 2013, pp. 1–1.
  22. T. O. Aydin, N. Stefanoski, S. Croci, M. Gross, and A. Smolic, “Temporally coherent local tone mapping of hdr video,” ACM Transactions on Graphics (TOG), vol. 33, no. 6, pp. 1–13, 2014.
  23. M. Kang, J. H. Lee, I. Choi, and M. H. Kim, “Real-time hdr video tone mapping using high efficiency video coding,” in 2019 IEEE International Conference on Image Processing (ICIP).   IEEE, 2019, pp. 4649–4653.
  24. A. Benoit, D. Alleysson, J. Herault, and P. L. Callet, “Spatio-temporal tone mapping operator based on a retina model,” in International Workshop on Computational Color Imaging.   Springer, 2009, pp. 12–22.
  25. E. Reinhard, T. Pouli, T. Kunkel, B. Long, A. Ballestad, and G. Damberg, “Calibrated image appearance reproduction,” ACM Transactions on Graphics (TOG), vol. 31, no. 6, pp. 1–11, 2012.
  26. Z. Khan, M. Khanna, and S. Raman, “Fhdr: Hdr image reconstruction from a single ldr image using feedback network,” in 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP).   IEEE, 2019, pp. 1–5.
  27. L. Wang and K.-J. Yoon, “Deep learning for hdr imaging: State-of-the-art and future trends,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 12, pp. 8874–8895, 2021.
  28. V. Patel, P. Shah, and S. Raman, “A generative adversarial network for tone mapping hdr images,” National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics. NCVPRIPG, pp. 220–231, 2018.
  29. N. Zhang, C. Wang, Y. Zhao, and R. Wang, “Deep tone mapping network in hsv color space,” in 2019 IEEE Visual Communications and Image Processing (VCIP), 2019, pp. 1–4.
  30. A. Rana, P. Singh, G. Valenzise, F. Dufaux, N. Komodakis, and A. Smolic, “Deep tone mapping operator for high dynamic range images,” IEEE Transactions on Image Processing, vol. 29, pp. 1285–1298, 2019.
  31. H. Yeganeh and Z. Wang, “Objective quality assessment of tone-mapped images,” IEEE Transactions on Image processing, vol. 22, no. 2, pp. 657–667, 2012.
  32. C. Le, J. Yan, Y. Fang, and K. Ma, “Perceptually optimized deep high-dynamic-range image tone mapping,” in 2021 International Conference on Virtual Reality and Visualization (ICVRV), 2021.
  33. R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouch, “Zonal brightness coherency for video tone mapping,” Signal Processing: Image Communication, vol. 29, no. 2, pp. 229–246, 2014.
  34. C. Lee, C. Lee, and C.-S. Kim, “Contrast enhancement based on layered difference representation of 2d histograms,” IEEE transactions on image processing, vol. 22, no. 12, pp. 5372–5384, 2013.
  35. X. Guo, “Lime: A method for low-light image enhancement,” in Proceedings of the 24th ACM international conference on Multimedia, 2016, pp. 87–91.
  36. M. Li, J. Liu, W. Yang, X. Sun, and Z. Guo, “Structure-revealing low-light image enhancement via robust retinex model,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2828–2841, 2018.
  37. X. Ren, W. Yang, W.-H. Cheng, and J. Liu, “Lr3m: Robust low-light enhancement via low-rank regularized retinex model,” IEEE Transactions on Image Processing, vol. 29, pp. 5862–5876, 2020.
  38. R. Wang, Q. Zhang, C.-W. Fu, X. Shen, W.-S. Zheng, and J. Jia, “Underexposed photo enhancement using deep illumination estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 6849–6857.
  39. Z. Zhang, H. Zheng, R. Hong, M. Xu, S. Yan, and M. Wang, “Deep color consistent network for low-light image enhancement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1899–1908.
  40. W. Yang, S. Wang, Y. Fang, Y. Wang, and J. Liu, “From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3063–3072.
  41. A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool, “Wespe: weakly supervised photo enhancer for digital cameras,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 691–700.
  42. C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zero-reference deep curve estimation for low-light image enhancement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1780–1789.
  43. C. Li, C. G. Guo, and C. C. Loy, “Learning to enhance low-light image via zero-reference deep curve estimation,” IEEE Transactions on pattern analysis and machine intelligence, 2021.
  44. Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang, “Enlightengan: Deep light enhancement without paired supervision,” IEEE Transactions on Image Processing, vol. 30, pp. 2340–2349, 2021.
  45. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  46. F. Lv, F. Lu, J. Wu, and C. Lim, “Mbllen: Low-light image/video enhancement using cnns.” in BMVC, vol. 220, no. 1, 2018, p. 4.
  47. F. Zhang, Y. Li, S. You, and Y. Fu, “Learning temporal consistency for low light video enhancement from single images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4967–4976.
  48. Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3733–3742.
  49. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
  50. T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, “Contrastive learning for unpaired image-to-image translation,” in European conference on computer vision.   Springer, 2020, pp. 319–345.
  51. L. Wang, Y. Wang, X. Dong, Q. Xu, J. Yang, W. An, and Y. Guo, “Unsupervised degradation representation learning for blind super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 581–10 590.
  52. H. Wu, Y. Qu, S. Lin, J. Zhou, R. Qiao, Z. Zhang, Y. Xie, and L. Ma, “Contrastive learning for compact single image dehazing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 551–10 560.
  53. X. Chen, J. Pan, K. Jiang, Y. Li, Y. Huang, C. Kong, L. Dai, and Z. Fan, “Unpaired deep image deraining using dual contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2017–2026.
  54. J. Han, M. Shoeiby, T. Malthus, E. Botha, J. Anstee, S. Anwar, R. Wei, L. Petersson, and M. A. Armin, “Single underwater image restoration by contrastive learning,” in 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS.   IEEE, 2021, pp. 2385–2388.
  55. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations, 2015.
  56. C. Chen, Q. Chen, M. N. Do, and V. Koltun, “Seeing motion in the dark,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3185–3194.
  57. K. Han, Y. Wang, J. Guo, Y. Tang, and E. Wu, “Vision gnn: An image is worth graph of nodes,” Advances in Neural Information Processing Systems, 2022.
  58. M. Tassano, J. Delon, and T. Veit, “Fastdvdnet: Towards real-time deep video denoising without flow estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1354–1363.
  59. C. Qi, J. Chen, X. Yang, and Q. Chen, “Real-time streaming video denoising with bidirectional buffers,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 2758–2766.
  60. D. Li, X. Shi, Y. Zhang, X. Wang, H. Qin, and H. Li, “No attention is needed: Grouped spatial-temporal shift for simple and efficient video restorers,” arXiv preprint arXiv:2206.10810, 2022.
  61. R. Fattal, D. Lischinski, and M. Werman, “Gradient domain high dynamic range compression,” in Proceedings of the 29th annual conference on Computer graphics and interactive techniques, 2002, pp. 249–256.
  62. C. Schlick, “Quantization techniques for visualization of high dynamic range pictures,” Photorealistic rendering techniques, vol. 20, 1994.
  63. J. Tumblin and G. Turk, “Lcis: A boundary hierarchy for detail-preserving contrast reduction,” in Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999, pp. 83–90.
  64. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  65. P. Schober, C. Boer, and L. A. Schwarte, “Correlation coefficients: appropriate use and interpretation,” Anesthesia & Analgesia, vol. 126, no. 5, pp. 1763–1768, 2018.
  66. X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2794–2802.
  67. N. Yu, G. Liu, A. Dundar, A. Tao, B. Catanzaro, L. S. Davis, and M. Fritz, “Dual contrastive loss and attention for gans,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6731–6742.
  68. E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 126–135.
  69. J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 2049–2062, 2018.
  70. G. Zaal, “Hdri haven dataset,” https://hdrihaven.com/, [Online; accessed 14-November-2020].
  71. J. Froehlich, S. Grandinetti, B. Eberhardt, S. Walter, A. Schilling, and H. Brendel, “Creating cinematic wide gamut hdr-video for the evaluation of tone mapping operators and hdr-displays,” in Digital photography X, vol. 9023.   SPIE, 2014, pp. 279–288.
  72. G. Eilertsen, J. Unger, and R. K. Mantiuk, “Evaluation of tone mapping operators for hdr video,” in High dynamic range video.   Elsevier, 2016, pp. 185–207.
  73. N. K. Kalantari, E. Shechtman, C. Barnes, S. Darabi, D. B. Goldman, and P. Sen, “Patch-based high dynamic range video.” ACM Trans. Graph., vol. 32, no. 6, pp. 202–1, 2013.
  74. J. Kronander, S. Gustavson, G. Bonnet, and J. Unger, “Unified hdr reconstruction from raw cfa data,” in IEEE international conference on computational photography (ICCP).   IEEE, 2013, pp. 1–9.
  75. G. Chen, C. Chen, S. Guo, Z. Liang, K.-Y. K. Wong, and L. Zhang, “Hdr video reconstruction: A coarse-to-fine network and a real-world benchmark dataset,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2502–2511.
  76. F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, “A benchmark dataset and evaluation methodology for video object segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 724–732.
  77. S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Transactions on Graphics (ToG), vol. 35, no. 6, pp. 1–12, 2016.
  78. W.-S. Lai, J.-B. Huang, O. Wang, E. Shechtman, E. Yumer, and M.-H. Yang, “Learning blind video temporal consistency,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 170–185.
  79. P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, “Deepflow: Large displacement optical flow with deep matching,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 1385–1392.
  80. S. Ferradans, M. Bertalmio, E. Provenzi, and V. Caselles, “An analysis of visual adaptation and contrast perception for tone mapping,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 10, pp. 2002–2012, 2011.
  81. Z. Mai, H. Mansour, R. Mantiuk, P. Nasiopoulos, R. Ward, and W. Heidrich, “Optimizing a tone curve for backward-compatible high dynamic range image and video compression,” IEEE transactions on image processing, vol. 20, no. 6, pp. 1558–1571, 2010.
  82. Q. Shan, J. Jia, and M. S. Brown, “Globally optimized linear windowed tone mapping,” IEEE transactions on visualization and computer graphics, vol. 16, no. 4, pp. 663–675, 2009.
  83. X.-S. Zhang, K. fu Yang, J. Zhou, and Y.-J. Li, “Retina inspired tone mapping method for high dynamic range images,” Opt. Express, vol. 28, no. 5, pp. 5953–5964, Mar 2020. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-28-5-5953
  84. I. R. Khan, W. Aziz, and S. Shim, “Tone-mapping using perceptual-quantizer and image histogram,” IEEE Access, vol. 8, pp. 31 350–31 358, 2020.
  85. K. Ma, H. Yeganeh, K. Zeng, and Z. Wang, “High dynamic range image compression by optimizing tone mapped image quality index,” IEEE Transactions on Image Processing, vol. 24, no. 10, pp. 3086–3097, 2015.
  86. S. Paris, S. W. Hasinoff, and J. Kautz, “Local laplacian filters: Edge-aware image processing with a laplacian pyramid,” ACM Trans. Graph., vol. 30, no. 4, 2011.
  87. K. Gu, S. Wang, G. Zhai, S. Ma, X. Yang, W. Lin, W. Zhang, and W. Gao, “Blind quality assessment of tone-mapped images via analysis of information, naturalness, and structure,” IEEE Transactions on Multimedia, vol. 18, no. 3, pp. 432–443, 2016.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Cong Cao (21 papers)
  2. Huanjing Yue (32 papers)
  3. Xin Liu (820 papers)
  4. Jingyu Yang (45 papers)
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - cao-cong/UnCLTMO: Unsupervised HDR Image and Video Tone Mapping via Contrastive Learning. TCSVT 2023 (10 stars)
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube