Tensor Ring Decomposition (1606.05535v1)

Abstract: Tensor networks have in recent years emerged as the powerful tools for solving the large-scale optimization problems. One of the most popular tensor network is tensor train (TT) decomposition that acts as the building blocks for the complicated tensor networks. However, the TT decomposition highly depends on permutations of tensor dimensions, due to its strictly sequential multilinear products over latent cores, which leads to difficulties in finding the optimal TT representation. In this paper, we introduce a fundamental tensor decomposition model to represent a large dimensional tensor by a circular multilinear products over a sequence of low dimensional cores, which can be graphically interpreted as a cyclic interconnection of 3rd-order tensors, and thus termed as tensor ring (TR) decomposition. The key advantage of TR model is the circular dimensional permutation invariance which is gained by employing the trace operation and treating the latent cores equivalently. TR model can be viewed as a linear combination of TT decompositions, thus obtaining the powerful and generalized representation abilities. For optimization of latent cores, we present four different algorithms based on the sequential SVDs, ALS scheme, and block-wise ALS techniques. Furthermore, the mathematical properties of TR model are investigated, which shows that the basic multilinear algebra can be performed efficiently by using TR representaions and the classical tensor decompositions can be conveniently transformed into the TR representation. Finally, the experiments on both synthetic signals and real-world datasets were conducted to evaluate the performance of different algorithms.

• The paper introduces the Tensor Ring model that extends Tensor Train decomposition by leveraging cyclic tensor cores for balanced rank distribution and permutation invariance.
• The paper proposes four optimization algorithms (TR-SVD, TR-ALS, TR-ALSAR, and TR-BALS) demonstrating tradeoffs between efficiency and adaptive rank determination.
• Experimental results on synthetic and real datasets validate that TR decomposition outperforms traditional techniques in compression and classification tasks.

Tensor Ring Decomposition: A Comprehensive Overview

The paper conducted by Zhao et al. introduces a novel tensor decomposition method called Tensor Ring (TR) decomposition, which extends the capacity of existing tensor network approaches such as Tensor Train (TT) decomposition. The key innovation behind TR decomposition is its ability to alleviate the limitations in TT by providing a more flexible model with circular multilinear products, allowing the ranks of the tensor to be more evenly distributed across its dimensions. The circular structure of TR decompositions not only facilitates more generalized and powerful representation capabilities but also enhances their robustness against dimensional permutations.

Tensor networks, and specifically tensor decompositions like TT, hold substantial promise for large-scale optimization problems by reducing the complexity of high-dimensional data into more manageable parts. The TT decomposition is known for its efficiency but suffers from limitations due to its strictly sequential multilinear operations. The paper by Zhao et al. proposes the TR model, which addresses these constraints by employing a cyclic interconnection of 3rd-order tensor cores, and thus, mathematically and graphically ensuring permutation invariance through the trace operation.

The authors propose four different algorithms to optimize the TR model: TR-SVD, TR-ALS, TR-ALSAR, and TR-BALS. Each of these algorithms provides a unique approach towards optimizing the latent cores:

1. TR-SVD: Based on a non-recursive sequential SVD, this method offers stability and efficiency for approximating arbitrary tensors. However, it might not always yield optimal TR-ranks due to its dependence on the initial mode selection.
2. TR-ALS: The Alternating Least Squares (ALS) method focuses on non-orthogonal approximations and provides a stable approach, assuming TR-ranks are predefined.
3. TR-ALSAR: This ALS variant adapts the TR-ranks during iterations, providing an intuitive and heuristic approach to achieving desired approximation errors, though the computational cost is high.
4. TR-BALS: This method combines the advantages of ALS with block-wise optimization and adapts ranks using truncated SVD, offering computational efficiency and optimal rank adaptation without the need for extensive initial rank specification.

The paper underscores the advantageous properties of TR representations, demonstrating efficient execution of tensor operations such as addition, multiplication, and inner product computations directly on core representations. This efficiency is crucial for handling large-scale tensor data where traditional methods may falter.

The relationship of TR with other tensor decomposition methods such as Canonical Polyadic (CP), Tucker, and TT models is noteworthy. The TR model is shown to be a generalization of TT, sharing structural similarities yet offering superior flexibility and representation power, especially in terms of rank distribution and dimension permutation invariance.

Through extensive experiments using synthetic data, as well as video and image datasets, the authors validate the practical significance of TR decompositions. In synthetic settings, TR demonstrated remarkable adaptability and resilience to noise in comparison to TT and CP formats. When applied to real datasets like COIL-100 and KTH video datasets, TR-based approaches showed superior performance in terms of both compression ability and classification accuracy, underscoring their effectiveness for representation learning and feature extraction tasks.

This paper contributes valuable insights to the field of tensor decompositions, particularly by expanding the intricate yet highly efficient representation abilities of tensor networks. Future research directions could focus on refining these optimization algorithms further, exploring semi-supervised or supervised augmentations, and integrating TR decompositions into wider machine learning frameworks for enhanced data representation and processing capabilities.