3D Face Morphable Models "In-the-Wild" (1701.05360v1)

Abstract: 3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and among the state-of-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions ("in-the-wild"). In this paper, we propose the first, to the best of our knowledge, "in-the-wild" 3DMM by combining a powerful statistical model of facial shape, which describes both identity and expression, with an "in-the-wild" texture model. We show that the employment of such an "in-the-wild" texture model greatly simplifies the fitting procedure, because there is no need to optimize with regards to the illumination parameters. Furthermore, we propose a new fast algorithm for fitting the 3DMM in arbitrary images. Finally, we have captured the first 3D facial database with relatively unconstrained conditions and report quantitative evaluations with state-of-the-art performance. Complementary qualitative reconstruction results are demonstrated on standard "in-the-wild" facial databases. An open source implementation of our technique is released as part of the Menpo Project.

• The paper introduces an innovative texture modeling approach that learns statistical representations of facial features from unconstrained images, eliminating the need for illumination parameter optimization.
• The paper presents a fast fitting algorithm based on advanced deformable model techniques, achieving state-of-the-art accuracy in dense vertex error and normal estimation.
• The paper offers an open-source implementation and a new quantitative dataset, enabling enhanced practical applications and further research in 3D facial reconstruction.

Overview of 3D Face Morphable Models "In-the-Wild"

The paper focuses on developing a robust methodology for 3D facial reconstruction from single images captured under unconstrained conditions, referred to as "in-the-wild". The authors introduce an innovative approach to building and fitting 3D Morphable Models (3DMMs) which integrates both a statistical model of facial shape, incorporating identity and expression variations, and an "in-the-wild" texture model. The latter addresses challenges of modeling facial textures in environments devoid of controlled conditions, simplifying the process by eliminating the necessity to optimize illumination parameters.

Key Contributions

The paper presents four primary contributions that significantly enhance the practicality and accuracy of 3D facial reconstruction in real-world scenarios:

1. Texture Model Development: A methodology for learning a statistical texture model from "in-the-wild" facial images is proposed, ensuring full correspondence with a statistical shape prior. This model leverages feature-based representations such as HOG and SIFT to facilitate facial texture analysis without the complexities of illumination optimization.
2. Optimization Algorithm: A novel, fast fitting algorithm for "in-the-wild" 3DMMs is introduced, built on recent advances in statistical deformable model fitting. The algorithm improves computational efficiency while maintaining robustness and has been made publicly accessible through the Menpo Project.
3. Quantitative Evaluation with New Dataset: A new dataset featuring 3D facial surfaces captured under relatively unconstrained conditions utilizing KinectFusion technology is compiled. This dataset allows for quantitative assessments of 3D face reconstruction techniques, demonstrating improved accuracy over traditional methods.
4. Open Source Implementation: An open-source implementation of the proposed technique is released, promoting further development and research within the community.

Experimental Evaluation

The authors conduct comprehensive evaluations on their newly developed dataset. The experiments reveal that by leveraging an "in-the-wild" approach, the proposed 3DMMs outperform classical models, especially in terms of dense vertex error measurement and facial surface normal estimation. Their model exhibits state-of-the-art performance in reconstructing detailed 3D facial shapes, providing more accurate results than several Shape-from-Shading techniques.

Implications and Future Work

From a theoretical perspective, this work pushes the boundaries of traditional 3D facial modeling by effectively addressing the drawbacks of illumination parameter optimization. Practically, it facilitates 3D facial reconstruction in real-world scenarios where image conditions are usually unregulated. The enhancements in texture modeling and optimized fitting algorithms pave the way for improved face recognition, customization in virtual avatars, and applications in augmented reality.

Future developments in AI will likely extend the capabilities of 3DMMs by incorporating more complex textures and adapting to dynamic environmental changes. Additionally, wider application across diverse datasets will be crucial for generalizing the model's effectiveness and pushing its limits in terms of realism and adaptability.

In summary, the paper presents significant advancements in utilizing 3DMMs for facial reconstruction in "in-the-wild" scenarios, demonstrating enhanced model accuracy and robustness without traditional constraints. This framework not only advances theoretical aspects of 3D facial modeling but also opens doors to practical applications across various computer vision domains.