Skip to main content
SpringerLink
Log in
Book cover

Italian Workshop on Neural Nets

WIRN 2017 2017: Neural Advances in Processing Nonlinear Dynamic Signals pp 119–129Cite as

Hierarchical Temporal Representation in Linear Reservoir Computing

  • Chapter
  • First Online:

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 102))

Abstract

Recently, studies on deep Reservoir Computing (RC) highlighted the role of layering in deep recurrent neural networks (RNNs). In this paper, the use of linear recurrent units allows us to bring more evidence on the intrinsic hierarchical temporal representation in deep RNNs through frequency analysis applied to the state signals. The potentiality of our approach is assessed on the class of Multiple Superimposed Oscillator tasks. Furthermore, our investigation provides useful insights to open a discussion on the main aspects that characterize the deep learning framework in the temporal domain.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Angelov, P., Sperduti, A.: Challenges in deep learning. In: Proceedings of the 24th European Symposium on Artificial Neural Networks (ESANN), pp. 489–495. i6doc.com (2016)

    Google Scholar 

  2. Čerňanskỳ, M., Tiňo, P.: Predictive modeling with echo state networks. Artif. Neural Netw ICANN 2008, 778–787 (2008)

    Google Scholar 

  3. Frigo, M., Johnson, S.G.: FFTW: An adaptive software architecture for the FFT. In: Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 3, pp. 1381–1384. IEEE (1998)

    Google Scholar 

  4. Gallicchio, C., Martin-Guerrero, J., Micheli, A., Soria-Olivas, E.: Randomized machine learning approaches: Recent developments and challenges. In: Proceedings of the 25th European Symposium on Artificial Neural Networks (ESANN), pp. 77–86. i6doc.com (2017)

    Google Scholar 

  5. Gallicchio, C., Micheli, A.: Deep reservoir computing: a critical analysis. In: Proceedings of the 24th European Symposium on Artificial Neural Networks (ESANN), pp. 497–502. i6doc.com (2016)

    Google Scholar 

  6. Gallicchio, C., Micheli, A.: Echo state property of deep reservoir computing networks. Cogn. Comput. 337–350 (2017). https://doi.org/10.1007/s12559-017-9461-9

    Article  Google Scholar 

  7. Gallicchio, C., Micheli, A., Pedrelli, L.: Deep reservoir computing: a critical experimental analysis. Neurocomputing 87–99 (2017). https://doi.org/10.1016/j.neucom.2016.12.089

    Article  Google Scholar 

  8. Gallicchio, C., Micheli, A., Silvestri, L.: Local Lyapunov Exponents of Deep RNN. In: Proceedings of the 25th European Symposium on Artificial Neural Networks (ESANN), pp. 559–564. i6doc.com (2017)

    Google Scholar 

  9. Hermans, M., Schrauwen, B.: Training and analysing deep recurrent neural networks. In: NIPS, pp. 190–198 (2013)

    Google Scholar 

  10. Hihi, S.E., Bengio, Y.: Hierarchical recurrent neural networks for long-term dependencies. In: NIPS, pp. 493–499 (1995)

    Google Scholar 

  11. Holzmann, G., Hauser, H.: Echo state networks with filter neurons and a delay & sum readout. Neural Netw. 23(2), 244–256 (2010)

    Article  Google Scholar 

  12. Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)

    Article  Google Scholar 

  13. Jaeger, H., Lukoševičius, M., Popovici, D., Siewert, U.: Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw. 20(3), 335–352 (2007)

    Article  Google Scholar 

  14. Koryakin, D., Lohmann, J., Butz, M.: Balanced echo state networks. Neural Netw. 36, 35–45 (2012)

    Article  Google Scholar 

  15. Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)

    Article  Google Scholar 

  16. Otte, S., Butz, M.V., Koryakin, D., Becker, F., Liwicki, M., Zell, A.: Optimizing recurrent reservoirs with neuro-evolution. Neurocomputing 192, 128–138 (2016)

    Article  Google Scholar 

  17. Pasa, L., Sperduti, A.: Pre-training of recurrent neural networks via linear autoencoders. In: Advances in Neural Information Processing Systems, pp. 3572–3580 (2014)

    Google Scholar 

  18. Pascanu, R., Gülçehre, Ç., Cho, K., Bengio, Y.: How to construct deep recurrent neural networks, pp. 1–13. arXiv preprint arXiv:1312.6026v5 (2014)

  19. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  20. Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by evolino. Neural Comput. 19(3), 757–779 (2007)

    Article  Google Scholar 

  21. Verstraeten, D., Schrauwen, B., d’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Netw. 20(3), 391–403 (2007)

    Article  Google Scholar 

  22. Wierstra, D., Gomez, F.J., Schmidhuber, J.: Modeling systems with internal state using evolino. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1795–1802. ACM (2005)

    Google Scholar 

  23. Xue, Y., Yang, L., Haykin, S.: Decoupled echo state networks with lateral inhibition. Neural Netw. 20(3), 365–376 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Pisa, Largo B. Pontecorvo 3, Pisa, Italy

    Claudio Gallicchio, Alessio Micheli & Luca Pedrelli

Authors
  1. Claudio Gallicchio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessio Micheli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luca Pedrelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claudio Gallicchio .

Editor information

Editors and Affiliations

  1. Dipartimento di Psicologia, Università della Campana Luigi Vanvitelli, Caserta, Italy

    Anna Esposito

  2. Fundació Tecnocampus, Pompeu Fabra University, Mataro, Barcelona, Spain

    Marcos Faundez-Zanuy

  3. Department of Civil, Environmental, Energy, and Material Engineering, University Mediterranea of Reggio Calabria, Reggio Calabria, Italy

    Francesco Carlo Morabito

  4. Laboratorio di Neuronica, Dipartimento Elettronica e Telecomunicazioni, Politecnico di Torino, Torino, Italy

    Eros Pasero

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gallicchio, C., Micheli, A., Pedrelli, L. (2019). Hierarchical Temporal Representation in Linear Reservoir Computing. In: Esposito, A., Faundez-Zanuy, M., Morabito, F., Pasero, E. (eds) Neural Advances in Processing Nonlinear Dynamic Signals. WIRN 2017 2017. Smart Innovation, Systems and Technologies, vol 102. Springer, Cham. https://doi.org/10.1007/978-3-319-95098-3_11

Download citation

Publish with us

Policies and ethics

Search

Navigation