Learning Formation of Physically-Based Face Attributes (2004.03458v2)

Abstract: Based on a combined data set of 4000 high resolution facial scans, we introduce a non-linear morphable face model, capable of producing multifarious face geometry of pore-level resolution, coupled with material attributes for use in physically-based rendering. We aim to maximize the variety of face identities, while increasing the robustness of correspondence between unique components, including middle-frequency geometry, albedo maps, specular intensity maps and high-frequency displacement details. Our deep learning based generative model learns to correlate albedo and geometry, which ensures the anatomical correctness of the generated assets. We demonstrate potential use of our generative model for novel identity generation, model fitting, interpolation, animation, high fidelity data visualization, and low-to-high resolution data domain transferring. We hope the release of this generative model will encourage further cooperation between all graphics, vision, and data focused professionals while demonstrating the cumulative value of every individual's complete biometric profile.

• The paper introduces a high-fidelity generative model for 3D face attributes using a dataset of 4000 scans to achieve pore-level geometric detail.
• The paper decouples facial identity and expression into separate feature spaces, enabling precise and independent control of each component.
• The paper employs GANs for effective high-resolution upscaling, with performance validated through metrics like FID and Inception Score.

Learning Formation of Physically-Based Face Attributes: A Summary

The paper "Learning Formation of Physically-Based Face Attributes" introduces a novel framework aimed at generating high-fidelity, physically-based 3D morphable models of human faces. This paper is situated at the intersection of computer vision and graphics, with considerable potential applications in media entertainment, biometric modeling, and forensics.

Key Contributions

The research focuses on a deep learning-driven approach to creating a generative model that represents detailed facial geometry and associated texture maps for physically-based rendering. The core contributions of this work include:

1. High Fidelity Generative Model: Utilizing a dataset composed of 4000 high-resolution facial scans, the paper presents a non-linear morphable face model that achieves pore-level geometric detail, incorporating physically-based rendering material attributes such as albedo, specular intensity, and displacement maps.
2. Identity and Expression Modeling: It uniquely decouples identity from expressions, enabling separate, low-dimensional feature spaces for each. This separation allows for independent manipulation, fostering greater control over the generated facial avatars.
3. Scaling and Upscaling: Leveraging generative adversarial networks (GANs), the model effectively synthesizes high-resolution (4K) texture maps from lower-resolution inputs, demonstrating a novel cascading strategy for data upsampling.

Methodology

The paper utilizes automated pipelines, beginning with initial scans captured through high-precision systems like Light Stage setups, which is followed by mesh processing to establish a uniform model topology. The generative model itself is composed of two sub-networks:

• Identity Network: Modelled on the Style-GAN architecture, it jointly generates both geometry and albedo, ensuring anatomical realism through a multi-discriminator framework.
• Expression Network: This component models expression offsets using a different generative mechanism, facilitated by an expression regression module to maintain controlled semantic expression variations.

Evaluation

The model's efficacy was corroborated through several qualitative and quantitative assessments. Importantly, the identity network's joint generation was validated using metrics like Frechet Inception Distance (FID) and Inception Score (IS), reflecting high plausibility and fidelity in the generative outputs.

Implications and Future Directions

The proposed framework presents a transformative step towards democratizing high-fidelity character modeling, which historically required significant manual labor and bespoke techniques. Its ability to produce high-resolution facial models with minimal manual intervention marks a substantial improvement over linear morphable models traditionally used in both academic and commercial contexts.

Future research could explore more sophisticated modeling of the correlations between identity and expression, as well as the potential to refine expression interpolation for enhanced realism. Additionally, further attempts to integrate hair, eyes, and other anatomical components within the generative model could lead to a holistic, comprehensive solution for digital human rendering.

In conclusion, this work provides a robust platform for real-time and offline rendering applications, setting a benchmark for future initiatives aimed at bridging the gap between high-fidelity digital character creation and efficient, scalable processes.