Unsupervised Image-to-Image Translation Networks (1703.00848v6)

Abstract: Unsupervised image-to-image translation aims at learning a joint distribution of images in different domains by using images from the marginal distributions in individual domains. Since there exists an infinite set of joint distributions that can arrive the given marginal distributions, one could infer nothing about the joint distribution from the marginal distributions without additional assumptions. To address the problem, we make a shared-latent space assumption and propose an unsupervised image-to-image translation framework based on Coupled GANs. We compare the proposed framework with competing approaches and present high quality image translation results on various challenging unsupervised image translation tasks, including street scene image translation, animal image translation, and face image translation. We also apply the proposed framework to domain adaptation and achieve state-of-the-art performance on benchmark datasets. Code and additional results are available in https://github.com/mingyuliutw/unit .

• The paper presents the UNIT framework that learns a joint distribution for image translation by enforcing a shared latent space using VAEs and GANs.
• It demonstrates state-of-the-art performance with a 90.53% accuracy in SVHN to MNIST domain adaptation and high-quality translations in varied tasks.
• The approach opens avenues for practical applications in fields like medical imaging and autonomous driving, while suggesting future enhancements for multimodal translations.

Unsupervised Image-to-Image Translation Networks: A Review

Summary

The paper by Ming-Yu Liu, Thomas Breuel, and Jan Kautz proposes a framework for unsupervised image-to-image translation, concentrating on the formidable task of learning a joint distribution of images from two different domains without paired examples. To address this, the authors introduce a shared-latent space assumption and incorporate a joint architecture based on Coupled Generative Adversarial Networks (CoGANs) and Variational Autoencoders (VAEs). The core idea is to align the high-level representations of images in both domains via the shared-latent space assumption, facilitating effective translation between domains.

Key Concepts and Methodology

The complexity of inferring joint distributions from marginal distributions in unsupervised settings is mitigated by assuming that corresponding images across different domains can be mapped to the same latent representation. The contributions of this paper are embodied in the proposed UNIT framework, which integrates VAEs for ensuring correspondence between input and translated images, and GANs for enforcing the realism of the translated images.

Framework Structure

The UNIT framework consists of six subnetworks:

1. Encoders (E1 and E2): Extract latent representations from two image domains.
2. Generators (G1 and G2): Generate images from the latent space.
3. Discriminators (D1 and D2): Distinguish real from generated images in their respective domains.

A weight-sharing constraint binds these networks, specifically in the latter layers of E1 and E2, and the initial layers of G1 and G2. This enforces the shared-latent space assumption. The framework also leverages cycle-consistency constraints, analogous to those in CycleGANs, which empirically enhance translation performance by ensuring that translating an image and then translating it back yields the original image.

Experiments and Results

The authors conducted a series of experiments to validate their framework, including street scene translation, animal image translation, face attribute translation, and domain adaptation tasks. Notable results include:

• Street scenes: High-quality translations between sunny, rainy, night, summer, and winter scenes.
• Animal translation: Effective transformations between dog breeds and cat species.
• Face attributes: Reliable adjustment of facial attributes such as adding glasses or making someone appear smiling.
• Domain adaptation: Achieved state-of-the-art results in adapting digit classifiers trained on one dataset (e.g., SVHN) to work on another (e.g., MNIST).

In detail, the UNIT framework outperformed existing methods in unsupervised domain adaptation tasks, with a remarkable accuracy of 90.53% for SVHN to MNIST adaptation, surpassing the previous best of 84.88%.

Implications and Future Directions

The UNIT framework demonstrates promising practical implications in scenarios demanding high-fidelity image translation without paired training data. This has potential applications in medical imaging, autonomous driving, and other fields requiring robust image transformations.

However, the model's unimodal nature, stemming from the Gaussian latent space assumption, is a limitation worth addressing. Future research may involve:

• Integrating more complex latent space distributions to enable multimodal translations.
• Risk mitigation strategies for the inherent instability in training GANs.
• Extending the framework to handle high-resolution image translation tasks more efficiently.

Conclusion

This paper offers an important contribution to the burgeoning field of unsupervised image-to-image translation. The UNIT framework combines the strengths of VAEs and GANs with innovative constraints to achieve impressive results across a diversity of tasks. By advancing the state-of-the-art in domain adaptation and image translation, this research opens new avenues for exploring scalable, efficient, and versatile unsupervised learning methodologies in image processing and beyond.