Learning Generative Models of Textured 3D Meshes from Real-World Images (2103.15627v2)

Abstract: Recent advances in differentiable rendering have sparked an interest in learning generative models of textured 3D meshes from image collections. These models natively disentangle pose and appearance, enable downstream applications in computer graphics, and improve the ability of generative models to understand the concept of image formation. Although there has been prior work on learning such models from collections of 2D images, these approaches require a delicate pose estimation step that exploits annotated keypoints, thereby restricting their applicability to a few specific datasets. In this work, we propose a GAN framework for generating textured triangle meshes without relying on such annotations. We show that the performance of our approach is on par with prior work that relies on ground-truth keypoints, and more importantly, we demonstrate the generality of our method by setting new baselines on a larger set of categories from ImageNet - for which keypoints are not available - without any class-specific hyperparameter tuning. We release our code at https://github.com/dariopavllo/textured-3d-gan

• The paper presents a keypoint-free methodology using GANs to generate textured 3D meshes from real-world images.
• It leverages differentiable rendering and semantic templates to disentangle pose from appearance without keypoint annotations.
• Evaluations on datasets like CUB and ImageNet demonstrate competitive FID scores and enhanced pose estimation compared to traditional methods.

Overview of "Learning Generative Models of Textured 3D Meshes from Real-World Images"

This paper presents a method for learning generative models of textured 3D meshes from collections of real-world images, without relying on keypoint annotations for pose estimation. It leverages recent advances in differentiable rendering to disentangle pose and appearance, allowing generative models to better understand the concept of image formation. Traditional methods typically require annotated keypoints, limiting their applicability to specific datasets. The authors propose a framework using GANs that generates textured triangle meshes without such annotations while maintaining performance comparable to keypoint-reliant methods.

Methodology

The proposed method involves several key components:

• Pose Estimation without Keypoints: The approach eliminates the need for keypoint annotations and instead uses a class-specific mesh template and a pretrained semi-supervised object detector. Camera poses are estimated by optimizing multiple hypotheses for each object using object mask silhouettes. Subsequently, semantic layouts are refined utilizing inferred semantic templates, enhancing pose accuracy and resolving ambiguities inherent in silhouette-only approaches.
• Semantic Templates and Ambiguity Resolution: The framework initially assumes a generic mesh template for each class and employs a closed-form optimization technique to infer semantic part segmentations for 3D meshes. This step is crucial for resolving pose ambiguities by incorporating part-level semantic information into the pose estimation process.
• Generative Model Training: The paper builds upon previous GAN architectures for textured 3D mesh generation. Textures and shapes are modeled in UV space using a convolutional generative adversarial network (CNN-GAN). The network is capable of generating textures from partially observed data by leveraging a masking strategy.

The authors rigorously evaluate pose estimation and generative modeling capabilities on well-known datasets such as CUB, Pascal3D+, and a subset of ImageNet categories. Pose estimation results, which are measured against structure-from-motion (SfM)-derived poses, demonstrate that semantic augmentations significantly improve pose accuracy and recall, particularly in ambiguously posed images.

Results and Discussion

The paper shows promising results in terms of Fréchet Inception Distance (FID) across several categories. Notably, it manifests superior performance in generating 3D models for real-world datasets, where keypoints are absent or infrequent. The authors introduce new baselines by successfully training generative models on diverse ImageNet categories without class-specific hyperparameter tuning, which previous studies using synthetic data or smaller, annotated datasets have not achieved. The approach's ability to learn single models for generating all observed classes showcases desired disentanglement properties, with style and lighting variations emerging in learned latent spaces.

The paper concludes by suggesting avenues for future work, such as exploring more complex articulated templates for deformable objects and strengthening semantic generation techniques. It extends generative models' flexibility and applicability by reducing the need for extensive manual annotations, which could be particularly impactful in industrial applications where large-scale annotated datasets are impractical.

Conclusion

This research signifies an important step in 3D generative modeling by presenting a keypoint-free methodology that expands applicability to a wider range of datasets, enabling novel image categories. While the current generative models are visually convincing and support various forms of downstream applications in graphics and AI, future efforts could focus on improving the robustness of semantic inference and pose estimations in more ambiguous settings or explore conditional generative models to control and edit 3D shapes creatively. This work offers practical insights and tools for the expansion of 3D generative models in academia and industry.