LatentMan: Generating Consistent Animated Characters using Image Diffusion Models (2312.07133v2)

Abstract: We propose a zero-shot approach for generating consistent videos of animated characters based on Text-to-Image (T2I) diffusion models. Existing Text-to-Video (T2V) methods are expensive to train and require large-scale video datasets to produce diverse characters and motions. At the same time, their zero-shot alternatives fail to produce temporally consistent videos with continuous motion. We strive to bridge this gap, and we introduce LatentMan, which leverages existing text-based motion diffusion models to generate diverse continuous motions to guide the T2I model. To boost the temporal consistency, we introduce the Spatial Latent Alignment module that exploits cross-frame dense correspondences that we compute to align the latents of the video frames. Furthermore, we propose Pixel-Wise Guidance to steer the diffusion process in a direction that minimizes visual discrepancies between frames. Our proposed approach outperforms existing zero-shot T2V approaches in generating videos of animated characters in terms of pixel-wise consistency and user preference. Project page https://abdo-eldesokey.github.io/latentman/.

• The paper introduces a zero-shot approach that synthesizes animated videos with significantly enhanced temporal consistency.
• It employs Spatial Latent Alignment and Pixel-Wise Guidance to reduce frame disparities, achieving a 10% improvement measured by Human MSE.
• The methodology leverages pre-trained text-to-image diffusion models and depth maps to generate lifelike animations in dynamic environments.

Zero-Shot Synthesis of Animated Characters

Introduction to Zero-Shot Video Synthesis

The generation of video content featuring animated characters based on textual descriptions has substantial value across multiple industries, including entertainment and virtual reality. Classical approaches to creating Text-to-Video (T2V) content rely on extensive training on large datasets which can be costly and computationally demanding. Overcoming these hurdles, a zero-shot approach has been introduced for the creation of animated characters without the need for dedicated training. This method builds upon pre-trained Text-to-Image (T2I) diffusion models, commonly used for still image generation, to produce temporally consistent video sequences.

Methodology for Consistency

In pursuit of temporal consistency—a challenge for zero-shot T2V generation—the proposed method engages a zero-shot learning paradigm, leveraging existing text-based motion diffusion models. A sequence of guidance signals derived from textual inputs directs the T2I model in video frame generation. Key to maintaining consistency is the Spatial Latent Alignment module, responsible for aligning the latent codes that represent various elements across video frames. The Pixel-Wise Guidance strategy refines this alignment, steering the diffusion process to minimize disparities across frames. Moreover, a novel metric called Human Mean Squared Error has been introduced for measuring temporal consistency.

Technical Framework

The texture and form of an animated character must remain coherent throughout a video sequence for the content to be perceived as lifelike and high-quality. To ensure this, the authors designed a process that renders human poses into depth maps, leveraged as guidance for a pre-trained diffusion model. Dense correspondences between frames are calculated to line up the latent spaces, where the video frames are generated. This meticulous alignment guards against the temporal inconsistencies previously noted in similar zero-shot approaches.

Results and Contributions

The approach has shown a clear advantage over existing methods, enhancing pixel-wise consistency and garnering stronger user preference in the authors' studies. It presents a significant step forward in the ability to render diverse animated characters performing complex movements within dynamic environments. The method's key contributions include the integrated Spatial Latent Alignment and Pixel-Wise Guidance and the introduction of the new Human Mean Squared Error metric demonstrating a 10% improvement in temporal consistency.

Limitations and Future Directions

The research acknowledges the method's limitations related to the reliance on depth conditioning and the challenges of achieving perfect cross-frame correspondences. These imperfections can sometimes result in textured inconsistencies within the generated video content. Despite these challenges, even the Spatial Latent Alignment component alone has shown remarkable improvements. Looking into the future, refining the cross-frame correspondences could bring more precise alignment, improving the realism and fidelity of the generated animated characters. Furthermore, integrating background dynamics could enhance the overall realism of the videos.