End-to-end optimization of nonlinear transform codes for perceptual quality (1607.05006v2)

Abstract: We introduce a general framework for end-to-end optimization of the rate--distortion performance of nonlinear transform codes assuming scalar quantization. The framework can be used to optimize any differentiable pair of analysis and synthesis transforms in combination with any differentiable perceptual metric. As an example, we consider a code built from a linear transform followed by a form of multi-dimensional local gain control. Distortion is measured with a state-of-the-art perceptual metric. When optimized over a large database of images, this representation offers substantial improvements in bitrate and perceptual appearance over fixed (DCT) codes, and over linear transform codes optimized for mean squared error.

• The paper introduces an end-to-end framework that optimizes nonlinear transform codes with a differentiable perceptual metric, significantly enhancing image compression quality.
• It employs Generalized Divisive Normalization (GDN) for the analysis transform and an approximated inverse for synthesis, replacing quantization with additive noise to enable gradient descent.
• Experimental results show that the proposed approach outperforms traditional linear methods in rate–distortion performance, especially at low bitrates as measured by a normalized Laplacian pyramid metric.

End-to-End Optimization of Nonlinear Transform Codes for Perceptual Quality

The paper presented by Johannes Ballé, Valero Laparra, and Eero P. Simoncelli proposes a comprehensive framework for optimizing nonlinear transform codes focused on perceptual quality. Their approach is premised on the idea that existing transform coding paradigms, primarily linear, may not sufficiently capture the nonlinear characteristics that affect both the rate-distortion performance and perceptual quality of image compression systems. The motivation stems from the inadequacy of traditional error metrics like mean squared error (MSE) to align with human perceptual evaluations.

Framework Overview

The key contribution of this paper lies in the introduction of a new framework that enables end-to-end optimization of nonlinear transform codes with scalar quantization. This is achieved by optimizing differentiable pairs of analysis and synthesis transforms in tandem with a perceptual metric that is also differentiable. The process involves applying a series of transformations that include an analysis transform $g_a$ and an overview transform $g_s$. The representation of images in this framework is transformed into a code domain, quantized, and then reconstructed.

The paper utilizes a perceptual metric based on the normalized Laplacian pyramid (NLP), a representation inspired by the human visual system's early stages, to measure distortion. This metric is chosen over standard metrics like PSNR, which are known to be misaligned with perceived visual quality. By employing a perceptual transform prior to distortion computation, the framework significantly improves visual quality predictions.

Methodology

The paper details the use of Generalized Divisive Normalization (GDN) as the analysis transform. GDN is suited for capturing non-linear statistical characteristics prevalent in natural images. The synthesis transform, crucial for reconstruction, is an approximation of GDN's inverse, optimized for efficient decoding.

To facilitate optimization, the paper innovatively replaces the non-differentiable quantization process with additive uniform noise, enabling the application of gradient-descent methods such as the Adam optimizer. This modified approach simulates the quantization process and retains the differential properties needed for back-propagation during the optimization of network parameters.

Results

Experimental results demonstrate substantial improvements in rate–distortion characteristics when using the proposed nonlinear framework, particularly when assessed by perceptual metrics. The GDN-based transforms, when optimized for the NLP perceptual metric, outperform traditional linear transforms such as the DCT even when advanced quantization methods are employed. This illustrates the proposed method's potential for achieving high compression ratios with better perceptual quality.

The visual quality evaluation on the Kodak image set exemplifies the system's proficiency at low bitrates, where traditional methods tend to overspend bits on high-contrast regions, failing to perceptually allocate resources effectively. Nonlinear transform codes optimized for perception offer a balanced rate allocation, enhancing overall visual quality.

Implications and Future Directions

The proposed methodology has profound implications for image compression technologies, particularly in settings where eventual perceived quality is paramount. By leveraging differentiable transformations and perceptually aligned distortion metrics, the framework sets a precedent for improved codec designs.

Future research can explore the integration of sophisticated perceptual models within this framework, further enhancing optimization capabilities. Additionally, exploring adaptive entropy coding and investigating more complex signal-adaptive nonlinear transforms could provide further refinements to this promising approach.

In conclusion, this paper presents a pivotal advance in nonlinear image compression, reframing the task from a perceptual quality perspective through end-to-end optimization. It opens avenues for further interdisciplinary research between the domains of machine learning, signal processing, and cognitive perception, with promising applications in various multimedia technologies.