Deep Learning without Weight Symmetry

An Overview of the Product Feedback Alignment Algorithm

The paper "Deep Learning without Weight Symmetry" by Ji-An Li and Marcus K. Benna introduces the Product Feedback Alignment (PFA) algorithm, addressing the notorious weight symmetry problem in backpropagation (BP). BP remains a cornerstone algorithm for training artificial neural networks but is often criticized for its lack of biological plausibility. Its requirement for symmetric weights in forward and backward connections is not observed in biological neural networks. The new algorithm proposed in this work seeks to overcome this limitation while maintaining performance comparable to BP.

Introduction and Motivation

Artificial neural networks, much like their biological counterparts, need to efficiently update synaptic weights to improve performance on tasks. The BP algorithm, though successful, is not biologically plausible due to the need for precisely symmetric weights in the forward and backward passes. Biological neural networks do not exhibit this symmetry, prompting the need for alternative learning rules. Previous attempts to address this issue, such as Feedback Alignment (FA) and Direct Feedback Alignment (DFA), have shown limited performance, especially in deep networks and convolutional layers.

Product Feedback Alignment (PFA) Algorithm

The PFA algorithm proposed in this study avoids explicit weight symmetry by introducing an additional population of neurons to align forward weights with the product of feedback weights. Mathematically, feedforward weights ( W ) align with the product of two feedback weights ( R ) and ( B ) (i.e., ( W \propto (RB)^T )). This innovation allows PFA to approximate the performance of BP closely, even in deeper networks and more complex tasks.

Empirical Results

The effectiveness of PFA is validated through various experiments, including training on the MNIST, CIFAR-10, and ImageNet datasets using different neural network architectures:

• MNIST Dataset: A two-hidden-layer feedforward network was trained. PFA achieved test accuracy comparable to BP, significantly outperforming FA and DFA. Metrics such as backward-forward weight alignment and weight norm ratio further confirmed that PFA approximates BP closely.
• CIFAR-10 Dataset: ResNet-20 was employed for this experiment. PFA maintained performance consistency with BP and SF, particularly excelling over FA and DFA in task accuracy and stability of error propagation.
• ImageNet Dataset: Training with ResNet-18, PFA's performance was near that of BP, surpassing SF, which struggled with this more complex dataset and architecture.

Sparse Connectivity

The authors explored PFA's robustness in scenarios with sparse connections, a feature typical in biological brains. While task performance degraded with increasing sparsity in FA, DFA, and SF, PFA demonstrated superior resilience, suggesting potential advantages over existing approaches under biologically realistic constraints.

Implications and Future Directions

The introduction of PFA provides meaningful insights into developing more biologically plausible learning algorithms for deep neural networks. By efficiently avoiding explicit weight symmetry, PFA promises practical applications in scenarios where biological realism is paramount, such as neuromorphic systems.

The theoretical implications suggest that alignment mechanisms leveraging additional neuronal populations can facilitate more complex learning tasks without the constraints of traditional BP. Future research may explore plasticity rules for feedback weight adjustment to reduce the expansion ratio in PFA, potentially enhancing its biological plausibility.

In summary, the PFA algorithm marks a significant step towards reconciling the performance of deep learning models with the constraints observed in biological neural networks. While challenges such as convolutional weight sharing and computational overhead remain, PFA's promise in handling sparse connectivity and its close approximation to BP in performance positions it as a substantive advancement in the quest for biologically plausible algorithms.