HDRT: Infrared Capture for HDR Imaging
(2406.05475)Abstract
Capturing real world lighting is a long standing challenge in imaging and most practical methods acquire High Dynamic Range (HDR) images by either fusing multiple exposures, or boosting the dynamic range of Standard Dynamic Range (SDR) images. Multiple exposure capture is problematic as it requires longer capture times which can often lead to ghosting problems. The main alternative, inverse tone mapping is an ill-defined problem that is especially challenging as single captured exposures usually contain clipped and quantized values, and are therefore missing substantial amounts of content. To alleviate this, we propose a new approach, High Dynamic Range Thermal (HDRT), for HDR acquisition using a separate, commonly available, thermal infrared (IR) sensor. We propose a novel deep neural method (HDRTNet) which combines IR and SDR content to generate HDR images. HDRTNet learns to exploit IR features linked to the RGB image and the IR-specific parameters are subsequently used in a dual branch method that fuses features at shallow layers. This produces an HDR image that is significantly superior to that generated using naive fusion approaches. To validate our method, we have created the first HDR and thermal dataset, and performed extensive experiments comparing HDRTNet with the state-of-the-art. We show substantial quantitative and qualitative quality improvements on both over- and under-exposed images, showing that our approach is robust to capturing in multiple different lighting conditions.
Overview
-
The paper introduces HDRTNet, a deep neural network designed to combine thermal infrared (IR) and standard dynamic range (SDR) inputs to reconstruct high dynamic range (HDR) images, addressing the limitations of traditional HDR methods such as ghosting artifacts and missing details.
-
HDRTNet uses a U-Net architecture for infrared feature extraction and combines it with SDR data in a separate HDR image reconstruction branch, utilizing various loss functions to enhance image quality. The paper also introduces the first dataset of aligned HDR and thermal images, comprising 10,000 images under different lighting conditions.
-
Experimental results demonstrate that HDRTNet outperforms several state-of-the-art HDR reconstruction methods in terms of pu-PSNR, pu-SSIM, and pu-VSI metrics. The paper also discusses practical applications, implications, and future research directions, such as handling high-resolution images and better hardware integration.
HDRT: Infrared Capture for HDR Imaging
The paper "HDRT: Infrared Capture for HDR Imaging" introduces a novel approach for High Dynamic Range (HDR) image acquisition by leveraging thermal infrared (IR) sensors. Traditional HDR methods either rely on multiple exposure fusion, which is prone to ghosting artifacts due to longer capture times, or inverse tone mapping (ITM), which tries to generate HDR images from a single Standard Dynamic Range (SDR) image, often leading to missing details. To circumvent these limitations, the authors propose HDRTNet, a deep neural network that fuses IR and SDR content to produce HDR images.
Methodology Overview
HDRTNet is designed to integrate information from an IR sensor and an SDR camera to reconstruct HDR images, effectively addressing the problems of overexposure and underexposure. The authors divide the HDRTNet architecture into two main components: infrared feature extraction and HDR image reconstruction.
Infrared Feature Extraction:
- The IR branch of HDRTNet uses a U-Net architecture aimed at converting IR images to RGB images. This allows for the extraction of thermal features that are complementary to RGB data.
- The loss function for the IR branch combines pixel loss (mean absolute error and cosine similarity) with perceptual loss using a pretrained VGG-19 network.
HDR Image Reconstruction:
- The HDR branch fuses the extracted IR features with SDR data. This is achieved by combining the IR features with RGB data at shallow layers of a U-Net structure to reconstruct HDR images.
- The HDR branch loss function includes pixel loss, perceptual loss, and adversarial loss to enhance the quality of the generated HDR images.
Dataset
To validate their approach, the authors introduce the first dataset of aligned HDR and thermal images. The dataset consists of 10,000 images captured under various lighting conditions to highlight the robustness of the proposed method.
Experimental Results
HDRTNet is compared against several state-of-the-art single-image HDR reconstruction methods including DrTMO, Deep Recursive HDRI, HDRTVNet, LaNet, HDRCNN, Deep-HDR Reconstruction, ICTCPNet, ExpandNet, and HDRUNet. The evaluation metrics include perceptually uniform peak signal-to-noise ratio (pu-PSNR), structural similarity index measure (pu-SSIM), and visual saliency index (pu-VSI).
The proposed method shows substantial improvements:
- HDRTNet achieves the highest scores in pu-PSNR, pu-SSIM, and pu-VSI across overexposed, underexposed, and all images categories.
- Qualitatively, HDRTNet significantly outperforms other methods in recovering details lost to overexposure and underexposure.
Ablation Studies and Practical Applications
The authors conduct ablation studies to underscore the importance of their proposed modules:
- Feature-level Fusion vs. Pixel-level Fusion: Feature-level fusion is shown to be more effective in extracting relevant information from IR images, reducing visual artifacts.
- Separate vs. Combined Training: Training the IR branch separately is crucial to avoid extracting visually irrelevant features, which could degrade the HDR output.
Moreover, the paper discusses adaptations for handling high-resolution images and practical issues like registration errors. Techniques like pixel unshuffle and smooth gradient loss are introduced to address these challenges.
Implications and Future Work
The implications of this research are multifaceted:
- Practical Applications: The integration of thermal imaging into the HDR pipeline opens new avenues for robust image acquisition in adverse lighting conditions, enhancing applications in surveillance, autonomous driving, and photography.
- Theoretical Advancement: The study provides a new direction in HDR imaging research, demonstrating that non-visible spectra can significantly improve visible spectrum tasks.
Future research could delve into improving the integration of IR information when lighting conditions are optimal or tackling scenarios where materials block IR transmission. Additionally, hardware integration of IR and RGB sensors could further streamline the capture process.
Conclusion
The paper presents HDRTNet as a compelling method for HDR imaging by effectively leveraging thermal IR data to reconstruct HDR images from single-exposure SDR inputs. The proposed approach demonstrates clear advantages over existing methods, especially under extreme exposure conditions. The creation of the first aligned HDR and thermal dataset also marks a significant contribution, facilitating further research in this domain.
Create an account to read this summary for free: