Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks (1809.00886v1)

Abstract: We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a difficult and arduous process. To reduce the amount of required labeled training data, we propose to leverage the appearance information embedded in unlabeled images of spatially varying materials to self-augment the training process. Starting from an initial approximative network obtained from a small set of labeled training pairs, we estimate provisional model parameters for each unlabeled training exemplar. Given this provisional reflectance estimate, we then synthesize a novel temporary labeled training pair by rendering the exact corresponding image under a new lighting condition. After refining the network using these additional training samples, we re-estimate the provisional model parameters for the unlabeled data and repeat the self-augmentation process until convergence. We demonstrate the efficacy of the proposed network structure on spatially varying wood, metals, and plastics, as well as thoroughly validate the effectiveness of the self-augmentation training process.

• The paper introduces SVBRDF-net, a convolutional neural network that estimates spatially varying surface reflectance parameters from a single photograph using a novel self-augmentation training strategy.
• The self-augmentation strategy leverages unlabeled data by iteratively refining estimated parameters through rendering synthetic images under new lighting conditions.
• Empirical results demonstrate the network's effectiveness in generating plausible surface appearances under novel lighting and reducing reliance on large labeled datasets for diverse materials.

Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks

The paper entitled "Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks" presents an innovative approach to the challenging problem of estimating spatially varying surface reflectance from a single image of a planar material sample captured under unknown natural lighting conditions. The authors introduce a convolutional neural network (CNN) framework, referred to as SVBRDF-net, designed to estimate surface reflectance parameters that are physically plausible rather than strictly accurate. A notable contribution is the introduction of a self-augmentation training strategy intended to circumvent the necessity of acquiring expansive labeled datasets, a typically arduous task.

The primary challenge addressed in this work is the ill-posed nature of estimating the spatially varying bidirectional reflectance distribution function (SVBRDF) from limited visual information. Traditional practices often require skilled artists to manually produce reflectance decompositions, which are time-consuming and limited in scalability. To overcome these challenges, the authors propose leveraging unlabeled data by using their CNN to estimate provisional reflectance parameters which are then refined through a process of rendering synthetic images under novel lighting conditions. This iterative self-augmentation process refines the CNN progressively using a combination of labeled and synthesized data until convergence is achieved.

The network architecture designed in this paper consists of separate structures for homogeneous specular and spatially varying diffuse components. The SVBRDF-net is tasked with estimating relative diffuse albedo, homogeneous specular albedo, roughness parameters, and surface normals. Noteworthy numerical results are highlighted by their effective application across diverse materials such as wood, metals, and plastics, which underpin the robust nature of both the SVBRDF-net and the self-augmentation process.

Empirical results demonstrate the SVBRDF-net's effectiveness in producing reflectance parameters that yield visualizations coherent with the input photographs under new lighting setups. Furthermore, the self-augmentation approach exhibited significant reductions in the requirement for labeled data, enhancing the scalability for new material classes without compromising the precision essential for plausible reflectance estimation.

The implications of this research are multifaceted. Practically, it offers a sophisticated tool for scalable surface appearance modeling which is highly applicable in fields such as virtual content creation and computer graphics. Theoretically, it paves the way for further exploration into self-augmenting learning paradigms, offering insights into refining CNNs with limited supervised data by coupling them with exact inverse processes such as rendering algorithms.

Future directions posited by the authors suggest extensions to the network's applicability beyond planar samples, potentially opening avenues to handle local illumination conditions and spatially varying specularities. Additionally, there exists an opportunity to formalize the theoretical boundaries and conditions conducive to self-augmentation, shedding light on the broader applicability of this training strategy beyond SVBRDF estimation.

In conclusion, this paper introduces a rigorous paper on surface appearance modeling using convolutional neural networks, addressing significant challenges through self-augmentation and setting the stage for future developments and applications within artificial intelligence and surface reflectance modeling domains.