Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Shadow Generation for Composite Image Using Diffusion model (2403.15234v1)

Published 22 Mar 2024 in cs.CV

Abstract: In the realm of image composition, generating realistic shadow for the inserted foreground remains a formidable challenge. Previous works have developed image-to-image translation models which are trained on paired training data. However, they are struggling to generate shadows with accurate shapes and intensities, hindered by data scarcity and inherent task complexity. In this paper, we resort to foundation model with rich prior knowledge of natural shadow images. Specifically, we first adapt ControlNet to our task and then propose intensity modulation modules to improve the shadow intensity. Moreover, we extend the small-scale DESOBA dataset to DESOBAv2 using a novel data acquisition pipeline. Experimental results on both DESOBA and DESOBAv2 datasets as well as real composite images demonstrate the superior capability of our model for shadow generation task. The dataset, code, and model are released at https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Realtime estimation of illumination direction for augmented reality on mobile devices. In CIC, 2012.
  2. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
  3. Dovenet: Deep image harmonization via domain verification. In CVPR, 2020.
  4. Bargainnet: Background-guided domain translation for image harmonization. In ICME, 2021.
  5. High-resolution image harmonization via collaborative dual transformations. In CVPR, 2022.
  6. Improving the harmony of the composite image by spatial-separated attention module. TIP, 2020.
  7. Deep parametric indoor lighting estimation. In ICCV, 2019.
  8. Shadowformer: Global context helps image shadow removal. In AAAI, 2023a.
  9. Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. In CVPR, 2023b.
  10. Deep residual learning for image recognition. In CVPR, 2016.
  11. Denoising diffusion probabilistic models. In NeurlPS, 2020.
  12. Shadow generation for composite image in real-world scenes. AAAI, 2022.
  13. Mask-shadowgan: Learning to remove shadows from unpaired data. In ICCV, 2019.
  14. Diffusion model for mural image inpainting. In ITOEC, 2023.
  15. Automatic scene inference for 3d object compositing. ACM TOG, 2014.
  16. Exposing photo manipulation from shading and shadows. ACM TOG, 2014.
  17. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
  18. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
  19. Illumination animating and editing in a single picture using scene structure estimation. Computers & Graphics, 82:53–64, 2019.
  20. St-gan: Spatial transformer generative adversarial networks for image compositing. In CVPR, 2018.
  21. Static scene illumination estimation from videos with applications. JCST, 32(3):430–442, 2017.
  22. Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In CVPR, 2020.
  23. Image inpainting for irregular holes using partial convolutions. In ECCV, 2018.
  24. Glyphdraw: Learning to draw chinese characters in image synthesis models coherently. CoRR, abs/2303.17870, 2023.
  25. Automatic shadow generation via exposure fusion. IEEE Transactions on Multimedia, 2023.
  26. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
  27. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. CoRR, abs/2302.08453, 2023.
  28. Making images real again: A comprehensive survey on deep image composition. CoRR, abs/2106.14490, 2021.
  29. Survey on diverse image inpainting using diffusion models. In PCEMS, 2023.
  30. Pytorch: An imperative style, high-performance deep learning library. NIPS, 32, 2019.
  31. Poisson image editing. In SIGGRAPH. 2003.
  32. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  33. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
  34. Ssn: Soft shadow network for image compositing. In CVPR, 2021.
  35. Controllable shadow generation using pixel height maps. In ECCV, 2022.
  36. Pixht-lab: Pixel height based light effect generation for image compositing. In CVPR, 2023.
  37. Denoising diffusion implicit models. CoRR, abs/2010.02502, 2020.
  38. Objectstitch: Generative object compositing. In CVPR, 2023.
  39. Shadow generation with decomposed mask prediction and attentive shadow filling. In AAAI, 2024.
  40. Deep image harmonization. In CVPR, 2017.
  41. Instance shadow detection with a single-stage detector. TPAMI, 2022.
  42. Gp-gan: Towards realistic high-resolution image blending. In ACM MM, 2019.
  43. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
  44. Paint by example: Exemplar-based image editing with diffusion models. In CVPR, 2023a.
  45. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In ACM MM, 2023b.
  46. Adaptive composition gan towards realistic image synthesis. CoRR, abs/1905.04693, 2019.
  47. Towards realistic 3d embedding via view alignment. CoRR, abs/2007.07066, 2020.
  48. Controlcom: Controllable image composition using diffusion model. arXiv preprint arXiv:2308.10040, 2023a.
  49. Deep image compositing. In WACV, 2021.
  50. All-weather deep outdoor lighting estimation. In CVPR, 2019a.
  51. Deep image blending. In WACV, 2020.
  52. Adding conditional control to text-to-image diffusion models. In ICCV, 2023b.
  53. Shadowgan: Shadow synthesis for virtual objects with conditional adversarial networks. Computational Visual Media, 5:105–115, 2019b.
  54. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In CVPR, 2020.
  55. Pluralistic image completion. In CVPR, 2019.
  56. Image inpainting with cascaded modulation gan and object-aware training. In ECCV, 2022.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit

  1. [2403.15234] Shadow Generation for Composite Image Using Diffusion model (1 point, 0 comments)