Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
60 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Physics-guided Shape-from-Template: Monocular Video Perception through Neural Surrogate Models (2311.12796v3)

Published 21 Nov 2023 in cs.CV and cs.LG

Abstract: 3D reconstruction of dynamic scenes is a long-standing problem in computer graphics and increasingly difficult the less information is available. Shape-from-Template (SfT) methods aim to reconstruct a template-based geometry from RGB images or video sequences, often leveraging just a single monocular camera without depth information, such as regular smartphone recordings. Unfortunately, existing reconstruction methods are either unphysical and noisy or slow in optimization. To solve this problem, we propose a novel SfT reconstruction algorithm for cloth using a pre-trained neural surrogate model that is fast to evaluate, stable, and produces smooth reconstructions due to a regularizing physics simulation. Differentiable rendering of the simulated mesh enables pixel-wise comparisons between the reconstruction and a target video sequence that can be used for a gradient-based optimization procedure to extract not only shape information but also physical parameters such as stretching, shearing, or bending stiffness of the cloth. This allows to retain a precise, stable, and smooth reconstructed geometry while reducing the runtime by a factor of 400-500 compared to $\phi$-SfT, a state-of-the-art physics-based SfT approach.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Large steps in cloth simulation. In Annual Conference on Computer Graphics and Interactive Techniques, page 43–54, 1998.
  2. Shape-from-template. IEEE TPAMI, 37(10):2099–2118, 2015.
  3. Pbns: Physically based neural simulator for unsupervised garment pose space deformation. CoRR, abs/2012.11310, 2020.
  4. Neural cloth simulation. ACM TOG, 41, 2022.
  5. Recovering non-rigid 3d shape from image streams. In CVPR, pages 690–696 vol.2, 2000.
  6. The isowarp: the template-based visual geometry of isometric surfaces. IJCV, 129(7):2194–2222, 2021.
  7. Yarn-level simulation of woven cloth. ACM TOG, 33(6), 2014.
  8. Diffpd: Differentiable projective dynamics. ACM TOG, 41(2), 2021.
  9. Learning-based bending stiffness parameter estimation by a drape tester. ACM TOG, 41(6), 2022.
  10. Multiscale meshgraphnets. In ICML Workshop, 2022.
  11. Texture-generic deep shape-from-template. IEEE Access, 9:75211–75230, 2021.
  12. Hdm-net: Monocular non-rigid 3d reconstruction with learned deformation model. In Virtual Reality and Augmented Reality, pages 51–72. Springer International Publishing, 2018.
  13. Fine-grained differentiable physics: a yarn-level model for fabrics. In ICLR, 2022.
  14. Hood: Hierarchical graphs for generalized modelling of clothing dynamics. In CVPR, pages 16965–16974, 2023.
  15. Neuralsim: Augmenting differentiable simulators with neural networks. In IEEE ICRA, pages 9474–9481, 2021.
  16. Neural implicit representations for physical parameter inference from a single video. In WACV, pages 2093–2103, 2023.
  17. Difftaichi: Differentiable programming for physical simulation. In ICLR, 2020.
  18. Pavel Iakubovskii. Segmentation models. https://github.com/qubvel/segmentation_models, 2019.
  19. Estimating cloth simulation parameters from a static drape using neural networks. IEEE Access, 8:195113–195121, 2020.
  20. ϕitalic-ϕ\phiitalic_ϕ-SfT: Shape-from-Template with a Physics-Based Deformation Model. In CVPR, pages 3948–3958, 2022.
  21. Neuralclothsim: Neural deformation fields meet the kirchhoff-love thin shell theory. arXiv:2308.12970, 2023.
  22. Adam: A method for stochastic optimization. In ICLR, 2014.
  23. Modular primitives for high-performance differentiable rendering. ACM TOG, 39(6), 2020.
  24. Efficient cloth simulation using miniature cloth and upscaling deep neural networks. CoRR, 2019.
  25. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM TOG, 42(1), 2022.
  26. Differentiable cloth simulation for inverse problems. In NeurIPS. Curran Associates, Inc., 2019.
  27. Miles Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https://github.com/nvidia/warp, 2022. NVIDIA GPU Technology Conference (GTC).
  28. Physical simulation layer for accurate 3d modeling. In CVPR, pages 13514–13523, 2022.
  29. Adaptive anisotropic remeshing for cloth simulation. ACM TOG, 31(6), 2012.
  30. Dense image registration and deformable surface reconstruction in presence of occlusions and minimal texture. In ICCV, 2015.
  31. Hierarchical cloth simulation using deep neural networks. In CGI, page 139–146. ACM, 2018.
  32. Local non-rigid structure-from-motion from diffeomorphic mappings. In CVPR, 2020.
  33. Learning mesh-based simulation with graph networks. In ICLR, 2020.
  34. Geometry-aware network for non-rigid shape prediction from a single view. In CVPR, 2018.
  35. Scalable differentiable physics for learning and control. In ICML, pages 7847–7856, 2020.
  36. How will it drape like? capturing fabric mechanics from depth images. Computer Graphics Forum, 42(2):149–160, 2023.
  37. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241. Springer International Publishing, 2015.
  38. Deformable surface tracking ambiguities. In CVPR, pages 1–8, 2007.
  39. Self-supervised collision handling via generative 3d garment models for virtual try-on. In CVPR, pages 11763–11773, 2021.
  40. Snug: Self-supervised neural dynamic garments. In CVPR, pages 8140–8150, 2022.
  41. Ismo-gan: Adversarial learning for monocular non-rigid 3d reconstruction. In CVPRW, 2019.
  42. Neural dense non-rigid structure from motion with latent space constraints. In ECCV, 2020.
  43. John M. Sullivan. Curves of Finite Total Curvature, pages 137–161. Birkhäuser Basel, Basel, 2008.
  44. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects. In IROS, pages 10828–10835, 2022.
  45. State of the art in dense monocular non-rigid 3d reconstruction. Comput. Graph. Forum, 42(2):485–520, 2023.
  46. 3d human pose estimation via intuitive physics. In CVPR, pages 4713–4725, 2023.
  47. Unsupervised deep learning of incompressible fluid dynamics. CoRR, abs/2006.08762, 2020.
  48. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In ICCV, pages 14928–14940, 2023.
  49. Ppr: Physically plausible reconstruction from monocular videos. In ICCV, pages 3914–3924, 2023.
  50. Learning-based cloth material recovery from video. In ICCV, 2017.
  51. Direct, dense, and deformable: Template-based non-rigid 3d reconstruction from rgb video. In ICCV, 2015.
  52. Physdiff: Physics-guided human motion diffusion model. In ICCV, pages 16010–16021, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - DavidStotko/Physics-guided-Shape-from-Template (13 stars)
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube