Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Anything in Any Scene: Photorealistic Video Object Insertion (2401.17509v1)

Published 30 Jan 2024 in cs.CV

Abstract: Realistic video simulation has shown significant potential across diverse applications, from virtual reality to film production. This is particularly true for scenarios where capturing videos in real-world settings is either impractical or expensive. Existing approaches in video simulation often fail to accurately model the lighting environment, represent the object geometry, or achieve high levels of photorealism. In this paper, we propose Anything in Any Scene, a novel and generic framework for realistic video simulation that seamlessly inserts any object into an existing dynamic video with a strong emphasis on physical realism. Our proposed general framework encompasses three key processes: 1) integrating a realistic object into a given scene video with proper placement to ensure geometric realism; 2) estimating the sky and environmental lighting distribution and simulating realistic shadows to enhance the light realism; 3) employing a style transfer network that refines the final video output to maximize photorealism. We experimentally demonstrate that Anything in Any Scene framework produces simulated videos of great geometric realism, lighting realism, and photorealism. By significantly mitigating the challenges associated with video data generation, our framework offers an efficient and cost-effective solution for acquiring high-quality videos. Furthermore, its applications extend well beyond video data augmentation, showing promising potential in virtual reality, video editing, and various other video-centric applications. Please check our project website https://anythinginanyscene.github.io for access to our project code and more high-resolution video results.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Wasserstein gan, 2017.
  2. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
  3. Patchtable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics (ToG), 34(4):1–10, 2015.
  4. Learning texture manifolds with the periodic spatial gan. arXiv preprint arXiv:1705.06566, 2017.
  5. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  6. Learning continuous exposure value representations for single-image hdr reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12990–13000, 2023.
  7. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7230–7240, 2021.
  8. Dovenet: Deep image harmonization via domain verification. In CVPR, 2020.
  9. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
  10. Object removal by exemplar-based inpainting. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., pages II–II. IEEE, 2003.
  11. Stytr2: Image style transfer with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11326–11336, 2022.
  12. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017.
  13. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, pages 1033–1038. IEEE, 1999.
  14. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
  15. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  16. GitHub. Ray tracing examples and tutorials, 2023. https://github.com/nvpro-samples/vk_raytracing_tutorial_KHR/tree/master [Accessed: (October 16, 2023)].
  17. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  18. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
  19. Scene completion using millions of photographs. ACM Transactions on Graphics (ToG), 26(3):4–es, 2007.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  21. Image analogies. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 557–570. 2023.
  22. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  23. Temporally coherent video harmonization using adversarial networks. IEEE Transactions on Image Processing, 29:214–224, 2019.
  24. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
  25. Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv:1611.08207, 2016.
  26. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.
  27. Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Transactions on Image Processing, 16(11):2649–2661, 2007.
  28. Graphcut textures: Image and video synthesis using graph cuts. Acm transactions on graphics (tog), 22(3):277–286, 2003.
  29. Inserting videos into videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10061–10070, 2019.
  30. Precomputed real-time texture synthesis with markovian generative adversarial networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 702–716. Springer, 2016.
  31. Coda: A real-world road corner case dataset for object detection in autonomous driving. arXiv preprint arXiv:2203.07724, 2022.
  32. Aads: Augmented autonomous driving simulation using data-driven algorithms. Science robotics, 4(28):eaaw0863, 2019.
  33. Painterly image harmonization using diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 233–241, 2023.
  34. World-consistent video-to-video synthesis. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, pages 359–378. Springer, 2020.
  35. One million scenes for autonomous driving: Once dataset. 2021.
  36. Beyond grand theft auto v for training, testing and enhancing deep learning in self driving cars. arXiv preprint arXiv:1712.01397, 2017.
  37. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
  38. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  39. QIU023. Guivideodisplayselector: A simple tkinter-based gui application for video comparison and selection., 2023. https://github.com/QIU023/GUIVideoDisplaySelector [Accessed: (November 8, 2023)].
  40. Playing for data: Ground truth from computer games. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 102–118. Springer, 2016.
  41. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  42. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234–3243, 2016.
  43. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  44. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830–2839, 2017.
  45. Minos: Multimodal indoor simulator for navigation in complex environments. arXiv preprint arXiv:1712.03931, 2017.
  46. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339–9347, 2019.
  47. Hdr map reconstruction from a single ldr sky panoramic image for outdoor illumination estimation. IEEE Access, 2023.
  48. SideFX. Unreal plug-in, 2023. https://www.sidefx.com/products/houdini-engine/plug-ins/unreal-plug-in/ [Accessed: (October 24, 2023)].
  49. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  50. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  51. Delicate textured mesh recovery from nerf via adaptive surface refinement. arXiv preprint arXiv:2303.02091, 2023.
  52. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535, 2018.
  53. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
  54. Vulkan. Vulkan, cross platform 3d graphics, 2023. https://www.vulkan.org/ [Accessed: (October 16, 2023)].
  55. Video-to-video synthesis. arXiv preprint arXiv:1808.06601, 2018.
  56. Few-shot video-to-video synthesis. arXiv preprint arXiv:1910.12713, 2019.
  57. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068–9079, 2018.
  58. Pandaset: Advanced sensor suite dataset for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3095–3101. IEEE, 2021.
  59. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6721–6729, 2017.
  60. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
  61. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505–5514, 2018.
  62. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (14)
  1. Chen Bai (13 papers)
  2. Zeman Shao (14 papers)
  3. Guoxiang Zhang (5 papers)
  4. Di Liang (21 papers)
  5. Jie Yang (516 papers)
  6. Zhuorui Zhang (2 papers)
  7. Yujian Guo (1 paper)
  8. Chengzhang Zhong (1 paper)
  9. Yiqiao Qiu (5 papers)
  10. Zhendong Wang (60 papers)
  11. Yichen Guan (1 paper)
  12. Xiaoyin Zheng (3 papers)
  13. Tao Wang (700 papers)
  14. Cheng Lu (70 papers)
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces a comprehensive framework for seamlessly integrating 3D objects into dynamic videos by accurately estimating geometry and lighting conditions.
  • It employs a style transfer network to refine visual artifacts, enhancing color consistency and reducing noise for improved photorealism.
  • Empirical results demonstrate significant progress with a FID score of 3.730 and a human evaluation score of 61.11%, underscoring the framework's realism.

Introduction

The field of video simulation for applications such as virtual reality and film production is advancing rapidly, particularly with the integration of objects into dynamic video environments. This integration must meet stringent standards of physical realism, which hinges on accurate geometric alignment, lighting harmony, and seamless photorealistic blending of inserted objects with existing video footage.

Framework Overview

The paper introduces "Anything in Any Scene," a comprehensive framework that champions the seamless combination of 3D objects in dynamic video settings, addressing the geometric, lighting, and visual authenticity that prior methodologies have struggled to achieve. The authors identify the necessity of considering the intricate complexities that come with outdoor environments and the complications in incorporating a variety of object classes.

A cornerstone of the framework is its ability to estimate environment lighting, including sky and environmental conditions, to yield realistic shadowing effects. The framework further extends its ingenuity through a style transfer network that refines visual artifacts, such as noise discrepancies or color imbalances, enhancing the integration of the inserted object into the video with heightened photorealism.

Numerical Results and Framework Applications

Empirical results validate the framework's superiority in achieving high degrees of geometric, lighting, and photorealistic realism. An impressive quantitative leap is indicated with the lowest FID score at 3.730 and the highest human score at 61.11%, affirming superior performance in video simulation realism. Further substantiation comes from its applications in perception algorithms, demonstrating its potential in augmenting datasets to improve the performance of object detection models.

The versatility of this framework facilitates the creation of large-scale, realistic video datasets for diverse domains, exemplifying an efficient and cost-effective method for video data augmentation. It addresses challenges such as long-tail distribution and successfully navigates the constraints of out-of-distribution exemplars.

Conclusion

The paper concludes by underscoring the pivotal role of the proposed framework in the innovation of video simulation technology. It's presented as a malleable substrate, open to future enhancements with improved models, and boons for emerging applications in various video-dependent fields. This work stands as a testament to the ongoing evolution in the fabrication of synthetic video content, where realism and practicality are paramount.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Anything in Any Scene: Photorealistic Video Object Insertion
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube