Zero-shot Pose Transfer for Unrigged Stylized 3D Characters (2306.00200v1)

Abstract: Transferring the pose of a reference avatar to stylized 3D characters of various shapes is a fundamental task in computer graphics. Existing methods either require the stylized characters to be rigged, or they use the stylized character in the desired pose as ground truth at training. We present a zero-shot approach that requires only the widely available deformed non-stylized avatars in training, and deforms stylized characters of significantly different shapes at inference. Classical methods achieve strong generalization by deforming the mesh at the triangle level, but this requires labelled correspondences. We leverage the power of local deformation, but without requiring explicit correspondence labels. We introduce a semi-supervised shape-understanding module to bypass the need for explicit correspondences at test time, and an implicit pose deformation module that deforms individual surface points to match the target pose. Furthermore, to encourage realistic and accurate deformation of stylized characters, we introduce an efficient volume-based test-time training procedure. Because it does not need rigging, nor the deformed stylized character at training time, our model generalizes to categories with scarce annotation, such as stylized quadrupeds. Extensive experiments demonstrate the effectiveness of the proposed method compared to the state-of-the-art approaches trained with comparable or more supervision. Our project page is available at https://jiashunwang.github.io/ZPT

• The paper introduces a zero-shot method that transfers poses to unrigged, stylized 3D characters without explicit rigging or training data.
• It employs a dual-module design with a shape understanding and a pose deformation module to capture detailed character features and accurate deformations.
• The approach demonstrates superior generalization and volume preservation in experiments, reducing deformation errors across diverse character models.

Overview of Zero-shot Pose Transfer for Unrigged Stylized 3D Characters

The paper "Zero-shot Pose Transfer for Unrigged Stylized 3D Characters" addresses a critical and challenging issue in computer graphics: the automatic pose transfer for stylized 3D characters which are not rigged. It introduces a zero-shot approach that effectively maps the pose of a reference avatar to these characters without requiring explicit rigging or being trained on the stylized characters in various poses. The authors highlight the limitations of existing methods, such as the necessity for manual rigging or the need for target pose data during training, and propose an innovative approach leveraging deformation techniques devoid of explicit correspondence labeling.

Methodological Contributions

1. Shape Understanding Module: The authors present a semi-supervised module that understands and predicts the shape of 3D characters without needing direct vertex-to-vertex correspondence. This module captures both the coarse-level part segmentation and the detailed shape code representation that assists in the deformation process.
2. Pose Deformation Module: This module operates with an implicit function model that predicts the deformation of surface points based on a shape code and a sample pose code. It enables the transfer of the entire pose to the character without necessitating local correspondence annotations.
3. Test-time Training Procedure: To ensure realistic pose transformations, a novel volume-based method is introduced. This procedure performs test-time training that adjusts the model for unseen stylized characters to maintain deformation authenticity and accuracy, particularly focusing on volume preservation of the character's parts during the pose transfer.

Results and Performance

The proposed method was evaluated against state-of-the-art approaches using extensive experiments on various datasets, including human-like and quadrupedal models. The quantitative measures, such as PMD (Point-wise Mesh Euclidean Distance), and qualitative assessments showcased superior generalization capabilities in pose transfer by achieving lower deformation errors and maintaining character integrity. Moreover, unlike other methods that may only work well with annotated datasets or specific geometric deformations, this technique demonstrated strong performance in scenarios with scarce annotations or significantly varied character shapes.

Implications and Future Directions

This research provides a substantial contribution to automating the animation and gaming character development processes by eliminating the need for extensive manual rigging and enabling effective pose transfer across a diverse range of character shapes. The zero-shot capability signifies the potential for applications in areas where pre-collected pose datasets are limited or costly. In terms of future development, this approach can be extended by exploring its integration with real-time animation systems or further refining the test-time training procedures to accommodate more complex character anatomies, like deformable hands and other appendices. Moreover, expanding the methodology to effectively work with even broader character categories could increase its applicability.

Overall, the paper presents a robust method of zero-shot pose transfer that facilitates efficient and flexible animation processes, paving the way for more automated solutions in character animation. It sets a foundation for future explorations into reducing the reliance on annotations and manual rigging in animated graphics while broadening the scope of practical applications in the creative arts and interactive entertainment industries.