Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
164 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Resistive memory-based zero-shot liquid state machine for multimodal event data learning (2307.00771v3)

Published 3 Jul 2023 in cs.ET

Abstract: The human brain is a complex spiking neural network (SNN), capable of learning multimodal signals in a zero-shot manner by generalizing existing knowledge. Remarkably, it maintains minimal power consumption through event-based signal propagation. However, replicating the human brain in neuromorphic hardware presents both hardware and software challenges. Hardware limitations, such as the slowdown of Moore's law and Von Neumann bottleneck, hinder the efficiency of digital computers. Additionally, SNNs are characterized by their software training complexities. To this end, we propose a hardware-software co-design on a 40 nm 256 Kb in-memory computing macro that physically integrates a fixed and random liquid state machine (LSM) SNN encoder with trainable artificial neural network (ANN) projections. We showcase the zero-shot LSM-based learning of multimodal events on the N-MNIST and N-TIDIGITS datasets, including visual and audio data association, as well as neural and visual data alignment for brain-machine interfaces. Our co-design achieves classification accuracy comparable to fully optimized software models, resulting in a 152.83 and 393.07-fold reduction in training costs compared to SOTA contrastive language-image pre-training (CLIP) and Prototypical networks, and a 23.34 and 160-fold improvement in energy efficiency compared to cutting-edge digital hardware, respectively. These proof-of-principle prototypes demonstrate zero-shot multimodal events learning capability for emerging efficient and compact neuromorphic hardware.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Liu, K. et al. An optoelectronic synapse based on α𝛼\alphaitalic_α-in2se3 with controllable temporal dynamics for multimode and multiscale reservoir computing. \JournalTitleNature Electronics 5, 761–773 (2022).
  2. Embodied neuromorphic intelligence. \JournalTitleNature communications 13, 1024 (2022).
  3. Christensen, D. V. et al. 2022 roadmap on neuromorphic computing and engineering. \JournalTitleNeuromorphic Computing and Engineering 2, 022501 (2022).
  4. Physics for neuromorphic computing. \JournalTitleNature Reviews Physics 2, 499–510 (2020).
  5. A 128×\times×128 120 db 15μ𝜇\muitalic_μs latency asynchronous temporal contrast vision sensor. \JournalTitleIEEE journal of solid-state circuits 43, 566–576 (2008).
  6. Asynchronous binaural spatial audition sensor with 2×\times×64×\times×4 channel output. \JournalTitleIEEE Transactions on Biomedical Circuits and Systems 8, 453–464, DOI: 10.1109/TBCAS.2013.2281834 (2014).
  7. Jiménez-Fernández, A. et al. A binaural neuromorphic auditory sensor for fpga: a spike signal processing approach. \JournalTitleIEEE transactions on neural networks and learning systems 28, 804–818 (2016).
  8. Choo, K. D. et al. Energy-efficient motion-triggered iot cmos image sensor with capacitor array-assisted charge-injection sar adc. \JournalTitleIEEE Journal of Solid-State Circuits 54, 2921–2931, DOI: 10.1109/JSSC.2019.2939664 (2019).
  9. Finateu, T. et al. 5.10 a 1280×720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86µm pixels, 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline. In 2020 IEEE International Solid- State Circuits Conference - (ISSCC), 112–114, DOI: 10.1109/ISSCC19947.2020.9063149 (2020).
  10. Hsu, T.-H. et al. A 0.8 v multimode vision sensor for motion and saliency detection with ping-pong pwm pixel. \JournalTitleIEEE Journal of Solid-State Circuits 56, 2516–2524, DOI: 10.1109/JSSC.2021.3075746 (2021).
  11. Gallego, G. et al. Event-based vision: A survey. \JournalTitleIEEE transactions on pattern analysis and machine intelligence 44, 154–180 (2020).
  12. A dynamic vision sensor with 1% temporal contrast sensitivity and in-pixel asynchronous delta modulator for event encoding. \JournalTitleIEEE Journal of Solid-State Circuits 50, 2149–2160 (2015).
  13. Neuromorphic sensory systems. \JournalTitleCurrent opinion in neurobiology 20, 288–295 (2010).
  14. Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. \JournalTitleNature 577, 641–646 (2020).
  15. In-memory computing with resistive switching devices. \JournalTitleNature electronics 1, 333–343 (2018).
  16. Spintronic memristor through spin-torque-induced magnetization motion. \JournalTitleIEEE electron device letters 30, 294–297 (2009).
  17. The future of electronics based on memristive systems. \JournalTitleNature electronics 1, 22–29 (2018).
  18. Karunaratne, G. et al. In-memory hyperdimensional computing. \JournalTitleNature Electronics 3, 327–337 (2020).
  19. A long short-term memory for ai applications in spike-based neuromorphic hardware. \JournalTitleNature Machine Intelligence 4, 467–479 (2022).
  20. Karunaratne, G. et al. Robust high-dimensional memory-augmented neural networks. \JournalTitleNature communications 12, 2468 (2021).
  21. Moon, J. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. \JournalTitleNature Electronics 2, 480–487 (2019).
  22. Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. \JournalTitleNature communications 8, 2204 (2017).
  23. Neuromorphic nanoelectronic materials. \JournalTitleNature nanotechnology 15, 517–528 (2020).
  24. Yu, S. Neuro-inspired computing with emerging nonvolatile memorys. \JournalTitleProceedings of the IEEE 106, 260–285 (2018).
  25. Communication lower bound in convolution accelerators. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), 529–541 (IEEE, 2020).
  26. Rao, M. et al. Thousands of conductance levels in memristors integrated on cmos. \JournalTitleNature 615, 823–829 (2023).
  27. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. \JournalTitleIEEE Signal Processing Magazine 36, 51–63 (2019).
  28. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. \JournalTitleFrontiers in neuroscience 11, 682 (2017).
  29. Theory and tools for the conversion of analog to spiking convolutional neural networks. \JournalTitlearXiv preprint arXiv:1612.04052 (2016).
  30. Wu, Y. et al. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, vol. 33, 1311–1318 (2019).
  31. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. \JournalTitleJournal of neuroscience 18, 10464–10472 (1998).
  32. Phenomenological models of synaptic plasticity based on spike timing. \JournalTitleBiological cybernetics 98, 459–478 (2008).
  33. Brown, T. et al. Language models are few-shot learners. \JournalTitleAdvances in neural information processing systems 33, 1877–1901 (2020).
  34. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. \JournalTitleICLR (2021).
  35. Pattern classification by memristive crossbar circuits using ex situ and in situ training. \JournalTitleNature communications 4, 2072 (2013).
  36. Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. \JournalTitleNature 521, 61–64 (2015).
  37. Yu, S. et al. Binary neural network with 16 mb rram macro chip for classification and online training. In 2016 IEEE International Electron Devices Meeting (IEDM), 16–2 (IEEE, 2016).
  38. Yao, P. et al. Face classification using electronic synapses. \JournalTitleNature communications 8, 15199 (2017).
  39. Sheridan, P. M. et al. Sparse coding with memristor networks. \JournalTitleNature nanotechnology 12, 784–789 (2017).
  40. Bayat, F. M. et al. Implementation of multilayer perceptron network with highly uniform passive memristive crossbar circuits. \JournalTitleNature communications 9, 2331 (2018).
  41. Hu, M. et al. Memristor-based analog computation and neural network classification with a dot product engine. \JournalTitleAdvanced materials (Deerfield Beach, Fla.) 30 (2018).
  42. Cai, F. et al. A fully integrated reprogrammable memristor–cmos system for efficient multiply–accumulate operations. \JournalTitleNature Electronics 2, 290–299 (2019).
  43. Duan, Q. et al. Spiking neurons with spatiotemporal dynamics and gain modulation for monolithically integrated memristive neural networks. \JournalTitleNature communications 11, 3399 (2020).
  44. Joshi, V. et al. Accurate deep neural network inference using computational phase-change memory. \JournalTitleNature communications 11, 2473 (2020).
  45. Xue, C.-X. et al. A cmos-integrated compute-in-memory macro based on resistive random-access memory for ai edge devices. \JournalTitleNature Electronics 4, 81–90 (2021).
  46. Liu, Z. et al. Neural signal analysis with memristor arrays towards high-efficiency brain–machine interfaces. \JournalTitleNature communications 11, 4234 (2020).
  47. Zhong, Y. et al. Dynamic memristor-based reservoir computing for high-efficiency temporal signal processing. \JournalTitleNature communications 12, 408 (2021).
  48. Milano, G. et al. In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks. \JournalTitleNature Materials 21, 195–202 (2022).
  49. Dalgaty, T. et al. In situ learning using intrinsic memristor variability via markov chain monte carlo sampling. \JournalTitleNature Electronics 4, 151–161 (2021).
  50. Real-time computing without stable states: A new framework for neural computation based on perturbations. \JournalTitleNeural computation 14, 2531–2560 (2002).
  51. Radford, A. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763 (PMLR, 2021).
  52. Converting static image datasets to spiking neuromorphic datasets using saccades. \JournalTitleFrontiers in neuroscience 9, 437 (2015).
  53. Feature representations for neuromorphic audio spike streams. \JournalTitleFrontiers in neuroscience 12, 23 (2018).
  54. Prototypical networks for few-shot learning. \JournalTitleAdvances in neural information processing systems 30 (2017).
  55. Abbott, L. F. Lapicque’s introduction of the integrate-and-fire model neuron (1907). \JournalTitleBrain research bulletin 50, 303–304 (1999).
  56. Jia, C. et al. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 4904–4916 (PMLR, 2021).
  57. Deep liquid state machines with neural plasticity for video activity recognition. \JournalTitleFrontiers in neuroscience 13, 686 (2019).
  58. A digital liquid state machine with biologically inspired learning and its application to speech recognition. \JournalTitleIEEE transactions on neural networks and learning systems 26, 2635–2649 (2015).
  59. Reinforcement learning with low-complexity liquid state machines. \JournalTitleFrontiers in Neuroscience 13, 883 (2019).
  60. Short-term plasticity in a liquid state machine biomimetic robot arm controller. In 2017 International Joint Conference on Neural Networks (IJCNN), 3399–3408 (IEEE, 2017).
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now