Fast Patch-based Style Transfer of Arbitrary Style (1612.04337v1)

Abstract: Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.

• The paper introduces the 'Style Swap', a patch-based operation that replaces complex multi-layer optimization with a rapid, single-layer solution.
• It integrates an inverse network to generate images through a feedforward pass, effectively handling a broad range of new style and content inputs.
• Quantitative experiments demonstrate reduced computational load and iteration count, highlighting the method's potential for real-time image and video processing.

Overview of "Fast Patch-based Style Transfer of Arbitrary Style"

This paper introduces a novel method for artistic style transfer, aimed at efficiently combining the content of one image with the style of another without being confined to a set number of trained styles. The authors propose a local matching approach that leverages patch-based style and content integration within a single layer of a pretrained CNN. The work identifies the drawbacks of traditional optimization-based methods that are computationally intensive and existing feedforward network methods that are limited by the number of preset styles. By addressing these limitations, this paper offers significant contributions to both the theory and practice of style transfer in computational vision.

The core innovation in this research is the introduction of a patch-based operation, termed "Style Swap," that effectively matches and swaps local patches between the content and style images using normalized cross-correlation. This operation is implemented using a series of traditional and transposed convolutional layers, facilitating a rapid and efficient style synthesis method. The "Style Swap" procedure replaces complex multi-layer optimization with a computationally straightforward single-layer solution, allowing users to stylize images consistently, including handling video frames in a coherent manner due to its consistency.

The authors also equip their style transfer framework with an "inverse network," trained to approximate the optimal image synthesis derived from patch-based transformed activations. This inverse network resolves the image generation task through a feedforward pass, enabling the system to adapt to an arbitrary range of new style and content inputs outside its training data.

The presented experiments demonstrate the expedited nature of the proposed method, achieving aesthetically appealing style transfer results rapidly compared to existing methods that either require extensive fine-tuning or repetitive training for different styles. Quantitatively, the optimization speed of their formulation shows a marked improvement, with fewer iterations necessary and a lower computational load per iteration compared to methods in prior art.

Implications and Future Directions

From a theoretical standpoint, the simplified objective and reduced optimization layers offer an intriguing direction for further exploration of learning paradigms that require less computational overhead, particularly in real-time applications. Practically, the technique holds promise in enhancing the capabilities of image and video processing applications where rapid style adaptation is essential, such as mobile applications or live video processing.

Future work in the field could potentially expand on incorporating additional global texture consistency measures and exploring inter-patch dependencies, both spatial and temporal, to further refine the quality of the style transfer. Additionally, improving the robustness of the network against a wider variety of input styles and developing mechanisms to effectively generalize style concepts across domains could lead to more powerful generative models in the domain of digital artistry.

In conclusion, the methodologies proposed in this work represent a substantial contribution to efficiently solving the artistic style transfer problem, remaining adaptable while ensuring performance suitable for practical applications. This research has succeeded in advancing the understanding and execution of style transfer tasks within the constraints of computational efficiency and versatility.