Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Contraction Properties of the Global Workspace Primitive (2310.01571v1)

Published 2 Oct 2023 in cs.LG, cs.SY, eess.SY, and q-bio.NC

Abstract: To push forward the important emerging research field surrounding multi-area recurrent neural networks (RNNs), we expand theoretically and empirically on the provably stable RNNs of RNNs introduced by Kozachkov et al. in "RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks". We prove relaxed stability conditions for salient special cases of this architecture, most notably for a global workspace modular structure. We then demonstrate empirical success for Global Workspace Sparse Combo Nets with a small number of trainable parameters, not only through strong overall test performance but also greater resilience to removal of individual subnetworks. These empirical results for the global workspace inter-area topology are contingent on stability preservation, highlighting the relevance of our theoretical work for enabling modular RNN success. Further, by exploring sparsity in the connectivity structure between different subnetwork modules more broadly, we improve the state of the art performance for stable RNNs on benchmark sequence processing tasks, thus underscoring the general utility of specialized graph structures for multi-area RNNs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Rnns of rnns: Recursive construction of stable assemblies of recurrent neural networks. Advances in Neural Information Processing Systems, 35:30512–30527, 2022.
  2. T. Sejnowski. The Deep Learning Revolution. MIT Press, 2018.
  3. Towards the next generation of recurrent network models for cognitive neuroscience. Current opinion in neurobiology, 70:182–192, 2021.
  4. Attention is all you need. NeurIPS, 2017.
  5. Stable recurrent models. arXiv preprint arXiv:1805.10369, 2019.
  6. Human problem solving, volume 104. Prentice-hall Englewood Cliffs, NJ, 1972.
  7. Bernard J Baars. A cognitive theory of consciousness. Cambridge University Press, 1988.
  8. A neuronal model of a global workspace in effortful cognitive tasks. Proceedings of the national Academy of Sciences, 95(24):14529–14534, 1998.
  9. The emergence of modularity in biological systems. Phys Life Rev, 8, 2011.
  10. Principles of neural science, volume 4. McGraw-hill New York, 2000.
  11. Cortical travelling waves: mechanisms and computational principles. Nature Reviews Neuroscience, 19, 2018.
  12. Conscious processing and the global neuronal workspace hypothesis. Neuron, 105(5):776–798, 2020.
  13. Brain-wide interactions between neural circuits. Current Opinion in Neurobiology, 65:iii–v, Dec 2020. ISSN 09594388. doi: 10.1016/j.conb.2020.12.012.
  14. Inferring brain-wide interactions using data-constrained recurrent neural network models. biorxiv, 2021. doi: 10.1101/2020.12.18.423348. URL https://www.biorxiv.org/content/early/2021/03/11/2020.12.18.423348.
  15. Cortical areas interact through a communication subspace. Neuron, 102(1):249–259, 2019.
  16. A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping. Proceedings of the national academy of sciences, 117(50):32124–32135, 2020.
  17. The theory of facilitated variation. Proceedings of the National Academy of Sciences, 104(1):8582–8589, 2007. doi: 10.1073/pnas.0701035104.
  18. Facilitated variation: How evolution learns from past environments to generalize to new environments. PLOS Computational Biology, 4(11), 2008.
  19. N. Kashtan and U. Alon. Spontaneous evolution of modularity and network motifs. PNAS, 102, 2005.
  20. Deep learning and the global workspace theory. Trends in Neurosciences, 44(9):692–704, 2021.
  21. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.01197, 2022.
  22. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
  23. Stimulus onset quenches neural variability: a widespread cortical phenomenon. Nature Neuroscience, 13:369–378, 2010.
  24. John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
  25. H Sebastian Seung. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences, 93(23):13339–13344, 1996.
  26. Balanced amplification: a new mechanism of selective amplification of neural activity patterns. Neuron, 61(4):635–648, 2009.
  27. Achieving stable dynamics in neural circuits. PLoS computational biology, 16(8):e1007659, 2020.
  28. Seizures as imbalanced up states: excitatory and inhibitory conductances during seizure-like events. Journal of Neurophysiology, 109:1296–1306, 2013.
  29. The missing link. Nature Physics, 8(7):512–513, 2012.
  30. M.W. Salter and B. Stevens. Microglia emerge as central players in brain disease. Nature Medicine, 23:1018–1027, 2017.
  31. J. Sakai. How synaptic pruning shapes neural wiring duringdevelopment and, possibly, in disease. PNAS, 117:16096–16099, 2020.
  32. On contraction analysis for non-linear systems. Automatica, 34(6):683–696, 1998.
  33. Modularity, evolution, and the binding problem: a view from stability theory. Neural Networks, 14(2):137–145, 2001.
  34. Jean-Jacques Slotine. Modular stability tools for distributed computation and control. Int. J. Adaptive Control and Signal Processing, 17(6), 2003.
  35. Mapping thalamic innervation to individual l2/3 pyramidal neurons and modeling their ‘readout’ of visual input. Nature Neuroscience, 26:470–480, 2023.
  36. Neural correlates of sparse coding and dimensionality reduction. PLoS Comput Biol, 15, 2019.
  37. Winning the lottery with neurobiology: faster learning on many cognitive tasks with fixed sparse rnns. arXiv preprint arXiv:2207.03523, 2022.
  38. Excitatory and inhibitory interactions in localized populations of model neurons. Biophysical Journal, 12(1):1–24, 1972. ISSN 0006-3495. doi: https://doi.org/10.1016/S0006-3495(72)86068-5.
  39. Hurwitz stability of metzler matrices. IEEE Transactions On Automatic Control, 55(6):1484–1487, 2010.
  40. Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. GMD Report, 148, 2001.
  41. C. Curto and K. Morrison. Graph rules for recurrent neural network dynamics: extended version. arXiv preprint arXiv:2301.12638, 2023.
  42. Jean Gallier et al. The schur complement and symmetric positive semidefinite (and definite) matrices (2019). URL https://www. cis. upenn. edu/jean/schur-comp. pdf, 2020.
  43. Michaela Ennis. Behavior quantification as the missing link between fields: Tools for digital psychiatry and their role in the future of neurobiology. arXiv preprint arXiv:2305.15385, 2023.
  44. Deep learning, volume 1. MIT Press, 2016.
  45. JB Hutchins and SW Barger. Why neurons die: cell death in the nervous system. The Anatomical Record, 253(3):79–90, 1998.
  46. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941, 2015.
  47. Weight agnostic neural networks. NeurIPS, 2019.
  48. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019.
  49. Antisymmetricrnn: A dynamical system view on recurrent neural networks. arXiv preprint arXiv:1902.09689, 2019.
  50. Learning longer-term dependencies in rnns with auxiliary losses. arXiv preprint arXiv:1803.00144v3, 2018.
  51. Lipschitz recurrent neural networks. ICLR, 2021.
  52. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611v1, 2022.
  53. Efficiently modeling long sequences with structured state spaces. ICLR, 2022.
  54. Trellis networks for sequence modeling. ICLR, 2019.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now