Learning Delays Through Gradients and Structure: Emergence of Spatiotemporal Patterns in Spiking Neural Networks (2407.18917v2)

Abstract: We present a Spiking Neural Network (SNN) model that incorporates learnable synaptic delays through two approaches: per-synapse delay learning via Dilated Convolutions with Learnable Spacings (DCLS) and a dynamic pruning strategy that also serves as a form of delay learning. In the latter approach, the network dynamically selects and prunes connections, optimizing the delays in sparse connectivity settings. We evaluate both approaches on the Raw Heidelberg Digits keyword spotting benchmark using Backpropagation Through Time with surrogate gradients. Our analysis of the spatio-temporal structure of synaptic interactions reveals that, after training, excitation and inhibition group together in space and time. Notably, the dynamic pruning approach, which employs DEEP R for connection removal and RigL for reconnection, not only preserves these spatio-temporal patterns but outperforms per-synapse delay learning in sparse networks. Our results demonstrate the potential of combining delay learning with dynamic pruning to develop efficient SNN models for temporal data processing. Moreover, the preservation of spatio-temporal dynamics throughout pruning and rewiring highlights the robustness of these features, providing a solid foundation for future neuromorphic computing applications.

• The paper reveals that combining learnable synaptic delays with dynamic pruning leads to the emergence of distinct spatiotemporal excitation and inhibition patterns.
• The methodology utilizes surrogate gradient backpropagation along with DEEP R and RigL strategies to optimize sparse connectivity while enforcing Dale’s Principle.
• Experimental results on the Raw Heidelberg Digits dataset show a competitive 94.5% test accuracy, highlighting the model’s efficient temporal data processing.

Spatio-temporal Structure of Excitation and Inhibition in Spiking Neural Networks

The paper under review presents insights into the emergence of relational dynamics within spiking neural networks (SNNs), incorporating learnable synaptic delays through Dilated Convolution with Learnable Spacings (DCLS). The manifestation of excitation and inhibition in spatio-temporal patterns forms a central theme, which is explored through both biologically plausible constraints and sparse connectivity frameworks.

The authors trained their SNN model on the Raw Heidelberg Digits benchmark dataset, employing Backpropagation Through Time with surrogate gradients. Analyzing the network post-training revealed that excitation and inhibition are organized both spatially and temporally, demonstrating the capacity of SNNs to manage temporal data effectively. They implemented a pioneering pruning strategy using DEEP R for connection removal and RigL for connection reintroduction. Pragmatically, this strategy sustains an optimal number of connections, rendering the model efficient in terms of computation and memory.

Remarkably, the research enforces Dale’s Principle in the network, stipulating that neurons must function solely as excitatory or inhibitory units, enhancing the biological realism of the model. The findings suggest that by integrating learnable delays, dynamic pruning, and biological constraints, one can achieve highly efficient SNN models adept at temporal data processing. Furthermore, the spatio-temporal patterns in excitation and inhibition persisted in the more constrained, biologically plausible model, underscoring their robustness against pruning and rewiring mechanisms.

Empirical results from experiments showed a test accuracy of 94.5% on the keyword spotting task, achieved by models both unconstrained and those adhering to Dale’s principle with sparse connectivity. The research further provides a quantitative measure of spatio-temporal patterning using Moran’s I, confirming the presence of autocorrelations despite the variable strength of patterns across data classes.

From a classification performance perspective, the sparse Dalean network maintained a competitive edge when connections were reduced. Notably, dynamic pruning proved critical for sustaining high performance, especially as networks transitioned from dense to sparse. Delay learning's efficacy was more pronounced when structure was non-adaptive, suggesting a possible interaction between structure learning and delay adjustments.

The implications of these findings extend into theoretical and practical domains. Theoretically, they offer insights into the emergent dynamics of neural computation within strictly constrained environments. Practically, the methods establish a foundation for advancing neuromorphic computing applications, particularly where energy efficiency and real-time processing are paramount.

In conclusion, this work paves the path for future research exploring optimization frameworks for SNNs, particularly under constraints mimicking biological systems. As neuromorphic hardware continues to evolve, these findings offer a significant contribution to developing more efficient and biologically inspired models for complex temporal data handling tasks.