2000 character limit reached
Shadow Generation for Composite Image Using Diffusion model (2403.15234v1)
Published 22 Mar 2024 in cs.CV
Abstract: In the realm of image composition, generating realistic shadow for the inserted foreground remains a formidable challenge. Previous works have developed image-to-image translation models which are trained on paired training data. However, they are struggling to generate shadows with accurate shapes and intensities, hindered by data scarcity and inherent task complexity. In this paper, we resort to foundation model with rich prior knowledge of natural shadow images. Specifically, we first adapt ControlNet to our task and then propose intensity modulation modules to improve the shadow intensity. Moreover, we extend the small-scale DESOBA dataset to DESOBAv2 using a novel data acquisition pipeline. Experimental results on both DESOBA and DESOBAv2 datasets as well as real composite images demonstrate the superior capability of our model for shadow generation task. The dataset, code, and model are released at https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2.
- Realtime estimation of illumination direction for augmented reality on mobile devices. In CIC, 2012.
- Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
- Dovenet: Deep image harmonization via domain verification. In CVPR, 2020.
- Bargainnet: Background-guided domain translation for image harmonization. In ICME, 2021.
- High-resolution image harmonization via collaborative dual transformations. In CVPR, 2022.
- Improving the harmony of the composite image by spatial-separated attention module. TIP, 2020.
- Deep parametric indoor lighting estimation. In ICCV, 2019.
- Shadowformer: Global context helps image shadow removal. In AAAI, 2023a.
- Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. In CVPR, 2023b.
- Deep residual learning for image recognition. In CVPR, 2016.
- Denoising diffusion probabilistic models. In NeurlPS, 2020.
- Shadow generation for composite image in real-world scenes. AAAI, 2022.
- Mask-shadowgan: Learning to remove shadows from unpaired data. In ICCV, 2019.
- Diffusion model for mural image inpainting. In ITOEC, 2023.
- Automatic scene inference for 3d object compositing. ACM TOG, 2014.
- Exposing photo manipulation from shading and shadows. ACM TOG, 2014.
- Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
- Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
- Illumination animating and editing in a single picture using scene structure estimation. Computers & Graphics, 82:53–64, 2019.
- St-gan: Spatial transformer generative adversarial networks for image compositing. In CVPR, 2018.
- Static scene illumination estimation from videos with applications. JCST, 32(3):430–442, 2017.
- Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In CVPR, 2020.
- Image inpainting for irregular holes using partial convolutions. In ECCV, 2018.
- Glyphdraw: Learning to draw chinese characters in image synthesis models coherently. CoRR, abs/2303.17870, 2023.
- Automatic shadow generation via exposure fusion. IEEE Transactions on Multimedia, 2023.
- V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
- T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. CoRR, abs/2302.08453, 2023.
- Making images real again: A comprehensive survey on deep image composition. CoRR, abs/2106.14490, 2021.
- Survey on diverse image inpainting using diffusion models. In PCEMS, 2023.
- Pytorch: An imperative style, high-performance deep learning library. NIPS, 32, 2019.
- Poisson image editing. In SIGGRAPH. 2003.
- High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
- U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
- Ssn: Soft shadow network for image compositing. In CVPR, 2021.
- Controllable shadow generation using pixel height maps. In ECCV, 2022.
- Pixht-lab: Pixel height based light effect generation for image compositing. In CVPR, 2023.
- Denoising diffusion implicit models. CoRR, abs/2010.02502, 2020.
- Objectstitch: Generative object compositing. In CVPR, 2023.
- Shadow generation with decomposed mask prediction and attentive shadow filling. In AAAI, 2024.
- Deep image harmonization. In CVPR, 2017.
- Instance shadow detection with a single-stage detector. TPAMI, 2022.
- Gp-gan: Towards realistic high-resolution image blending. In ACM MM, 2019.
- Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
- Paint by example: Exemplar-based image editing with diffusion models. In CVPR, 2023a.
- Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In ACM MM, 2023b.
- Adaptive composition gan towards realistic image synthesis. CoRR, abs/1905.04693, 2019.
- Towards realistic 3d embedding via view alignment. CoRR, abs/2007.07066, 2020.
- Controlcom: Controllable image composition using diffusion model. arXiv preprint arXiv:2308.10040, 2023a.
- Deep image compositing. In WACV, 2021.
- All-weather deep outdoor lighting estimation. In CVPR, 2019a.
- Deep image blending. In WACV, 2020.
- Adding conditional control to text-to-image diffusion models. In ICCV, 2023b.
- Shadowgan: Shadow synthesis for virtual objects with conditional adversarial networks. Computational Visual Media, 5:105–115, 2019b.
- Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In CVPR, 2020.
- Pluralistic image completion. In CVPR, 2019.
- Image inpainting with cascaded modulation gan and object-aware training. In ECCV, 2022.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.