Linking connectivity, dynamics and computations in low-rank recurrent neural networks
(1711.09672)Abstract
Large scale neural recordings have established that the transformation of sensory stimuli into motor outputs relies on low-dimensional dynamics at the population level, while individual neurons exhibit complex selectivity. Understanding how low-dimensional computations on mixed, distributed representations emerge from the structure of the recurrent connectivity and inputs to cortical networks is a major challenge. Here, we study a class of recurrent network models in which the connectivity is a sum of a random part and a minimal, low-dimensional structure. We show that, in such networks, the dynamics are low dimensional and can be directly inferred from connectivity using a geometrical approach. We exploit this understanding to determine minimal connectivity required to implement specific computations, and find that the dynamical range and computational capacity quickly increase with the dimensionality of the connectivity structure. This framework produces testable experimental predictions for the relationship between connectivity, low-dimensional dynamics and computational features of recorded neurons.
Overview
-
The paper explores the link between synaptic connectivity, neural dynamics, and computational capacities in both biological and artificial neural networks by focusing on recurrent neural networks (RNNs) characterized by a hybrid connectivity matrix comprising random and low-rank structured components.
-
A geometrical framework, called the geometrical mean-field approach, is introduced to predict the low-dimensional dynamics of the network, enabling a theoretical understanding without extensive simulations. This framework helps in designing minimal connectivity structures for specific computational tasks.
-
Increasing the rank of the structured connectivity part enhances the network's dynamical and computational capacities, allowing it to handle more complex tasks. These models replicate key experimental observations in neuroscience, showing how structured connectivity can modulate both individual neuron responses and network-wide dynamics.
Linking connectivity, dynamics and computations in low-rank recurrent neural networks
This paper addresses the foundational problem of understanding the relationship between synaptic connectivity, neural dynamics, and computational capabilities within the context of biological and artificial neural networks. The authors study a specific class of recurrent neural network (RNN) models characterized by a connectivity matrix that is the sum of a random component and a structured, low-rank part. This hybrid connectivity structure aims to exemplify essential traits of cortical networks, where connectivity is neither fully random nor fully structured.
Main Contributions
Low-Dimensional Dynamics Emergence: The paper demonstrates that the interaction between the low-rank structured part and the random part of the connectivity yields low-dimensional activity patterns. Specifically, the dynamics can be characterized by a few dominant dimensions corresponding to the directions defined by the connectivity structure. This insight is crucial for bridging the gap between high-dimensional connectivity and the observed low-dimensional neural activity in the brain.
Geometric Framework for Connectivity-Induced Dynamics: The authors introduce a geometrical mean-field approach to predict the low-dimensional dynamics of the network. This involves analyzing the relationships among the connectivity vectors and external inputs. The framework provides explicit equations for macroscopic quantities such as the overlap between network activity and connectivity vectors. Notably, this predictive model bypasses the need for extensive simulations by offering analytical insights into how structured connectivity impacts network dynamics.
Design of Minimal Connectivity for Specific Computations: Utilizing their theoretical framework, the authors design minimal low-rank connectivity structures to implement specific computational tasks. They explore tasks such as basic binary discrimination, noisy stimulus detection, and context-dependent evidence integration. For instance, a unit-rank connectivity structure enables a network to perform Go-Nogo discrimination by aligning the right-connectivity vector with the readout and the left-connectivity vector with the input representing the Go stimulus.
Scalable Complexity with Rank-Increasing Connectivity: The paper highlights that increasing the rank of the structured part of the connectivity enhances the network's dynamical range and computational capacity. A rank-two structure, for instance, allows the network to perform more complex tasks requiring contextual modulation, such as context-dependent Go-Nogo discrimination and evidence integration. This property demonstrates the scalability of the proposed approach in handling more sophisticated real-world computational problems.
Results and Performance
The proposed models effectively reproduce several key experimental observations from neuroscience. For instance, the authors show that individual neuron responses in their models display high heterogeneity and mixed selectivity, akin to neural activity recorded in cortical areas. Furthermore, their framework captures the low-dimensional nature of neural dynamics, which scales with task complexity—a principle highlighted in biological studies.
The summary results for specific tasks include:
- Basic Binary Discrimination: The network successfully discriminates between a Go and a Nogo stimulus, exhibiting distinct two-dimensional activity in response to the Go stimulus and one-dimensional activity for the Nogo stimulus.
- Noisy Detection Task: For a noisy stimulus input, the network demonstrates a clear threshold behavior for detection, governed by the structured connectivity's overlap.
- Context-Dependent Discrimination and Integration: The network implements context-dependent tasks by modulating the effective threshold via contextual inputs, reflecting cognitive flexibility observed in biological systems.
Implications and Future Directions
The findings have profound implications for both theoretical neuroscience and the development of efficient artificial neural networks (ANNs). The low-rank recurrent network framework offers a unified conceptual approach to understanding how connectivity structures influence neural dynamics and computations. Its relevance extends to optimizing ANNs for specific tasks by judiciously designing their connectivity structures.
This work also opens up several avenues for future research. From a practical standpoint, extending the framework to spiking networks or incorporating biophysical constraints such as excitatory-inhibitory segregation could enhance the biological plausibility and functionality of the model. Theoretically, exploring higher-rank structures or alternative low-rank configurations may yield new insights into network scalability and capacity.
In conclusion, this paper contributes significantly to our understanding of the interplay between connectivity, dynamics, and computation in neural networks. By leveraging low-rank structures, it provides a robust framework for elucidating low-dimensional neural dynamics and designing efficient recurrent networks for complex computational tasks.
Create an account to read this summary for free: