Practical Lossless Compression with Latent Variables using Bits Back Coding

Bits Back with ANS: An Advanced Scheme for Lossless Compression Using Latent Variable Models

The paper “Bits Back with ANS (BB-ANS)” introduces a novel methodology to achieve lossless compression utilizing deep latent variable models. This scheme focuses on efficiently compressing data by integrating Asymmetric Numeral Systems (ANS) with bits back coding in the context of Variational Auto-Encoders (VAEs), providing compelling results, particularly with the MNIST dataset.

Overview and Background

The intrinsic connection between probabilistic modeling and lossless compression, underpinned by Shannon’s Source Coding Theorem, is well-established. Traditional algorithms like Huffman coding, arithmetic coding (AC), and the more sophisticated ANS are implementations of this relationship. However, the focus of these methods has mainly been on fully observed models, leaving latent variable models less explored in the realm of lossless compression.

Bits back coding initially emerged as a theoretical construct but has found practical application in some limited cases. However, integrating bits back coding with modern neural networks, particularly those with continuous latent variables, has been a challenge due to compatibility issues with AC, where model complexity and compression rate efficiency suffer. The BB-ANS approach overcomes these challenges by leveraging the stack-like nature of ANS, facilitating efficient and practical implementation of bits back coding.

Methodology: Bits Back with ANS (BB-ANS)

The core innovation in BB-ANS lies in the effective utilization of the stack-like properties of ANS to implement bits back coding without the rate overheads associated with AC. This scheme is designed to operate efficiently with modern neural network-based models, accommodating continuous latent variables by mapping continuous distributions into discrete approximations for practical encoding and decoding using ANS.

Detailed Process

1. Initial Setup: The sender starts with extra bits and the symbol to be encoded, decoding the extra bits to sample a latent variable from an approximate posterior ( q(y|s) ).
2. Encoding: Using the latent variable, they encode the symbol and the latent itself according to their respective priors, appending these bits back to the ANS stack in a manner that facilitates zero overhead for subsequent iterations.
3. Chaining: By continuously using the state of the ANS stack from encoding the previous sample as the starting state for the next, a daisy-chain-like process is created, enhancing overall efficiency.

Experimental results, applied to both binarized and full MNIST datasets, substantiate the efficacy of BB-ANS. When applied with a VAE, the system not only achieves but often surpasses the compression rates of standard algorithms (e.g., PNG, bz2), despite the simplicity of the VAE model architecture employed. The reported compression rates closely align with the negative ELBO values, suggesting minimal overhead and high fidelity of the scheme.

Results and Implications

The paper demonstrates that BB-ANS achieves:

• Superior Compression Rates: In compressing the MNIST dataset, BB-ANS outperforms traditional compression methods, achieving 0.19 bits/dimension for binarized MNIST and 1.41 bits/dimension for full MNIST in alignment with the negative ELBO.
• Practical Implementation: The straightforward implementation allows for efficient compression and decompression, suitable for large-scale datasets.
• Scalability: The reduction in complexity and the potential for parallelization make BB-ANS a promising candidate for real-world applications beyond small datasets like MNIST.

Future Developments

The research suggests several directions for future work:

1. Advanced Model Integration: Extending BB-ANS to state-of-the-art models such as PixelVAE suggests potentially significant improvements in compression rates, especially for larger and more complex datasets like ImageNet.
2. Parallelization: Incorporating GPU-based parallelization techniques could vastly improve the efficiency, making the compression process suitable for high-speed applications.
3. Broader Applications: Exploring the applicability of BB-ANS to time series models and other latent variable models could further broaden its utility.
4. Model Communication Efficiency: Amortizing the communication cost of model parameters over large datasets, or leveraging model compression techniques, could enhance the practicality of BB-ANS in various domains.

Conclusion

BB-ANS presents a robust and efficient method for lossless compression using latent variable models. By innovatively merging bits back coding with ANS, it overcomes prior limitations related to arithmetic coding, offering superior performance and paving the way for future enhancements in compression technology leveraging deep learning models. This work stands as a significant progression in utilizing latent variable models for practical, scalable, and efficient data compression.