Benjamin Chamand
ABSTRACT
The self-supervised learning (SSL) paradigm is a rapidly growing research area in recent years with promising results, especially in the field of image processing. In order for these models to converge towards the creation of discriminative representations, a data augmentation is applied to the input data that feeds two-branch networks. On the other hand, Spiking Neural Networks (SNNs) are attracting a growing community due to their ability to process temporal information, their low-energy consumption and their high biological plausibility. Thanks to the use of Poisson process stochasticity to encode the same data into different temporal representations, and the success of using surrogate gradient on learning, we propose a self-supervised learning method applied to an SNN network, and we make a preliminary study on the generated representations. We have shown its feasibility by training our architecture on a dataset of images of digits (MNIST), then we have evaluated the representations with two classification methods.
- Yoshua Bengio, Nicholas Léonard, and Aaron C Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. CoRR abs/1308.3432(2013).Google Scholar
- Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In 37th International Conference on Machine Learning, ICML 2020, Vol. PartF168147-3.Google ScholarDigital Library
- Xinlei Chen and Kaiming He. 2021. Exploring simple Siamese representation learning. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR46437.2021.01549Google ScholarCross Ref
- Xiang Cheng, Yunzhe Hao, Jiaming Xu, and Bo Xu. 2020. LISNN: Improving Spiking Neural Networks with Lateral Interactions for Robust Object Recognition. In IJCAI International Joint Conference on Artificial Intelligence, Vol. 2021-January. https://doi.org/10.24963/ijcai.2020/211Google Scholar
- Peter U. Diehl and Matthew Cook. 2015. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Frontiers in Computational Neuroscience 9, AUGUST (2015). https://doi.org/10.3389/fncom.2015.00099Google Scholar
- Jean Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap your own latent a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, Vol. 2020-December.Google Scholar
- Jesse Hagenaars, Frederico Paredes-Vallés, and Guido de Croon. 2021. Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks. In Advances in Neural Information Processing Systems.Google Scholar
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR42600.2020.00975Google ScholarCross Ref
- A. L. Hodgkin and A. F. Huxley. 1952. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology 117, 4 (1952). https://doi.org/10.1113/jphysiol.1952.sp004764Google ScholarCross Ref
- Saeed Reza Kheradpisheh, Mohammad Ganjtabesh, Simon J. Thorpe, and Timothée Masquelier. 2018. STDP-based spiking deep convolutional neural networks for object recognition. Neural Networks 99(2018). https://doi.org/10.1016/j.neunet.2017.12.005Google ScholarDigital Library
- Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.Google Scholar
- Yann LeCun, Corinna Cortes, and CJ Burges. 2010. MNIST handwritten digit database.Google Scholar
- Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. 2020. Spike-FlowNet: Event-Based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 12374 LNCS. https://doi.org/10.1007/978-3-030-58526-6_22Google ScholarDigital Library
- Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. 2020. Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures. Frontiers in Neuroscience 14 (2020). https://doi.org/10.3389/fnins.2020.00119Google Scholar
- Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. 2019. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine 36, 6 (2019). https://doi.org/10.1109/MSP.2019.2931595Google ScholarCross Ref
- Federico Paredes-Valles, Kirk Yannick Willehm Scheper, and Guido C.H.E. De Croon. 2020. Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8(2020). https://doi.org/10.1109/TPAMI.2019.2903179Google ScholarDigital Library
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, Vol. 32. 8026–8037.Google ScholarDigital Library
- Hélène Paugam-Moisy and Sander Bohte. 2012. Computing with Spiking Neuron Networks. In Handbook of Natural Computing. Vol. 1-4. https://doi.org/10.1007/978-3-540-92910-9_10Google Scholar
- Christian Pehle and Jens Egholm Pedersen. 2021. Norse - A deep learning library for spiking neural networks.Google Scholar
- Michael Pfeiffer and Thomas Pfeil. 2018. Deep Learning With Spiking Neurons: Opportunities and Challenges. https://doi.org/10.3389/fnins.2018.00774Google Scholar
- Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. 2019. Towards spike-based machine intelligence with neuromorphic computing. Nature 575, 7784 (2019). https://doi.org/10.1038/s41586-019-1677-2Google Scholar
- Sumit Bam Shrestha and Garrick Orchard. 2018. Slayer: Spike Layer Error Reassignment in Time. In Advances in Neural Information Processing Systems, Vol. 2018-December.Google Scholar
- Aboozar Taherkhani, Ammar Belatreche, Yuhua Li, Georgina Cosma, Liam P. Maguire, and T. M. McGinnity. 2020. A review of learning in biologically plausible spiking neural networks. Neural Networks 122(2020), 253–272. https://doi.org/10.1016/j.neunet.2019.09.036Google ScholarDigital Library
- Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. 2019. Deep Learning in Spiking Neural Networks. Neural Networks 111(2019), 47–63. https://doi.org/10.1016/j.neunet.2018.12.002Google ScholarDigital Library
- Simon Thorpe, Arnaud Delorme, and Rufin Van Rullen. 2001. Spike-based strategies for rapid processing. Neural Networks 14, 6-7 (2001). https://doi.org/10.1016/S0893-6080(01)00083-1Google ScholarCross Ref
- Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. 2018. Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks. Frontiers in Neuroscience 12, MAY (2018). https://doi.org/10.3389/fnins.2018.00331Google Scholar
- Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018. Unsupervised Feature Learning via Non-parametric Instance Discrimination. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2018.00393Google ScholarCross Ref
- Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12310–12320. https://proceedings.mlr.press/v139/zbontar21a.htmlGoogle Scholar
- Friedemann Zenke and Surya Ganguli. 2018. SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks. Neural Computation 30, 6 (2018). https://doi.org/10.1162/neco_a_01086Google ScholarDigital Library
- Friedemann Zenke and Tim P. Vogels. 2021. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks. Neural computation 33, 4 (2021). https://doi.org/10.1162/neco_a_01367Google Scholar
Index Terms
- Self-Supervised Spiking Neural Networks applied to Digit Classification
Recommendations
Supervised Learning of Logical Operations in Layered Spiking Neural Networks with Spike Train Encoding
Few algorithms for supervised training of spiking neural networks exist that can deal with patterns of multiple spikes, and their computational properties are largely unexplored. We demonstrate in a set of simulations that the ReSuMe learning algorithm ...
Supervised learning in multilayer spiking neural networks with inner products of spike trains
Recent advances in neurosciences have revealed that neural information in the brain is encoded through precisely timed spike trains, not only through the neural firing rate. This paper presents a new supervised, multi-spike learning algorithm for ...
Sparse and burst spiking in artificial neural networks inspired by synaptic retrograde signaling
The bursting of action potential and sparse activity are ubiquitously observed in the brain. Although the functions of these activity modes remain to be understood, it is expected that they play a critical role in information processing. In addition, ...
Comments