Skip to main content
SpringerLink
Log in

The EsnTorch Library: Efficient Implementation of Transformer-Based Echo State Networks

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1794))

Included in the following conference series:

  • 658 Accesses

Abstract

Transformer-based models have revolutionized NLP. But in general, these models are highly resource consuming. Based on this consideration, several reservoir computing approaches to NLP have shown promising results. In this context, we propose EsnTorch, a library that implements echo state networks (ESNs) with transformer-based embeddings for text classification. EsnTorch is developed in PyTorch, optimized to work on GPU, and compatible with the transformers and datasets libraries from Hugging Face: the major data science platform for NLP. Accordingly, our library can make use of all the models and datasets available from Hugging Face. A transformer-based ESN implemented in EsnTorch consists of four building blocks: (1) An embedding layer, which uses a transformer-based model to embed the input texts; (2) A reservoir layer, which can implements three kinds of reservoirs: recurrent, linear or null; (3) A pooling layer, which offers three kinds of pooling strategies: mean, last, or None; (4) And a learning algorithm block, which provides six different supervised learning algorithms. Overall, this work falls within the context of sustainable models for NLP.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The spectral radius of a matrix \(\textbf{W}\), denoted by \(\rho (\textbf{W})\), is the largest absolute value of the eigenvalues of \(\textbf{W}\).

References

  1. Cabessa, J., Hernault, H., Kim, H., Lamonato, Y., Levy, Y.Z.: Efficient text classification with echo state networks. In: International Joint Conference on Neural Networks, IJCNN 2021, pp. 1–8. IEEE (2021)

    Google Scholar 

  2. Cabessa, J., Lamonato, H.H.Y., Levy, Y.Z.: Combining bert and echo state networks for efficient text classification. Applied Intelligence (Submitted 2022)

    Google Scholar 

  3. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, Volume 1, 2019. pp. 4171–4186. ACL (2019)

    Google Scholar 

  4. Di Sarli, D., Gallicchio, C., Micheli, A.: Question classification with untrained recurrent embeddings. In: Alviano, M., Greco, G., Scarcello, F. (eds.) AI*IA 2019. LNCS (LNAI), vol. 11946, pp. 362–375. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-35166-3_26

    Chapter  Google Scholar 

  5. Dominey, P.F., Hoen, M., Inui, T.: A neurolinguistic model of grammatical construction processing. J. Cogn. Neurosci. 18(12), 2088–2107 (2006)

    Article  Google Scholar 

  6. Gallicchio, C., Micheli, A., Pedrelli, L.: Design of deep echo state networks. Neural Netw. 108, 33–47 (2018)

    Article  Google Scholar 

  7. Gandhi, M., Jaeger, H.: Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 25(3), 671–696 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  8. Hinaut, X., Dominey, P.F.: Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network simulation study using reservoir computing. PLOS ONE 8(2), 1–18 (2013)

    Google Scholar 

  9. Hinaut, X., Lance, F., Droin, C., Petit, M., Pointeau, G., Dominey, P.F.: Corticostriatal response selection in sentence production: Insights from neural network simulation with reservoir computing. Brain Lang. 150, 54–68 (2015)

    Article  Google Scholar 

  10. Hinaut, X., Petit, M., Pointeau, G., Dominey, P.F.: Exploring the acquisition and production of grammatical constructions through human-robot interaction with echo state networks. Front. Neurorobot. 8, 16 (2014)

    Article  Google Scholar 

  11. Hinaut, X., Twiefel, J.: Teach your robot your language! trainable neural parser for modeling human sentence processing: Examples for 15 languages. IEEE Trans. Cogn. Dev. Syst. 12(2), 179–188 (2020)

    Article  Google Scholar 

  12. Jaeger, H.: Short term memory in echo state networks. GMD-Report 152, GMD - German National Research Institute for Computer Science (2002)

    Google Scholar 

  13. Jaeger, H.: Echo state network. Scholarpedia 2(9), 2330 (2007)

    Google Scholar 

  14. Jaeger, H.: The "echo state" approach to analysing and training recurrent neural networks. GMD Report 148, GMD - German National Research Institute for Computer Science (2001)

    Google Scholar 

  15. Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)

    Article  Google Scholar 

  16. Lhoest, Q., et al.: Datasets: A community library for natural language processing. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175–184. ACL (2021)

    Google Scholar 

  17. Lukoševičius, M.: A practical guide to applying echo state networks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 659–686. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_36

    Chapter  Google Scholar 

  18. Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)

    Article  MATH  Google Scholar 

  19. Ramamurthy, R., Stenzel, R., Sifa, R., Ladi, A., Bauckhage, C.: Echo state networks for named entity recognition. In: Tetko, I.V., Kůrková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11731, pp. 110–120. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30493-5_11

    Chapter  Google Scholar 

  20. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108 (2019)

    Google Scholar 

  21. Schaetti, N.: Echotorch: Reservoir computing with pytorch. https://github.com/nschaetti/EchoTorch (2018)

  22. Schaetti, N.: Behaviors of reservoir computing models for textual documents classification. In: International Joint Conference on Neural Networks, IJCNN 2019 Budapest, Hungary, July 14–19, 2019, pp. 1–7. IEEE (2019)

    Google Scholar 

  23. Shen, S., Baevski, A., Morcos, A., Keutzer, K., Auli, M., Kiela, D.: Reservoir transformers. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4294–4309. ACL, Online (2021)

    Google Scholar 

  24. Shrivastava, H., Garg, A., Cao, Y., Zhang, Y., Sainath, T.N.: Echo state speech recognition. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, pp. 5669–5673. IEEE (2021)

    Google Scholar 

  25. Steiner, P., Jalalvand, A., Stone, S., Birkholz, P.: Pyrcn: A toolbox for exploration and application of reservoir computing networks (2021)

    Google Scholar 

  26. Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., Zhou, D.: Mobilebert: a compact task-agnostic BERT for resource-limited devices. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pp. 2158–2170. ACL (2020)

    Google Scholar 

  27. Tong, M.H., Bickett, A.D., Christiansen, E.M., Cottrell, G.W.: Learning grammatical structure with echo state networks. Neural Netw. 20(3), 424–432 (2007)

    Article  MATH  Google Scholar 

  28. Trouvain, N., Pedrelli, L., Dinh, T.T., Hinaut, X.: ReservoirPy: an efficient and user-friendly library to design echo state networks. In: Farkaš, I., Masulli, P., Wermter, S. (eds.) ICANN 2020. LNCS, vol. 12397, pp. 494–505. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61616-8_40

    Chapter  Google Scholar 

  29. Vaswani, A., et al: Attention is all you need. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 5998–6008 (2017)

    Google Scholar 

  30. Wieting, J., Kiela, D.: No training required: Exploring random encoders for sentence classification. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net (2019)

    Google Scholar 

  31. Wolf, T., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45. ACL, Online (202,

    Google Scholar 

  32. Yildiz, I.B., Jaeger, H., Kiebel, S.J.: Re-visiting the echo state property. Neural Netw. 35, 1–9 (2012)

    Article  MATH  Google Scholar 

Download references

Acknowledgment

The authors are grateful to Playtika Ltd. for contributing to an inspiring R &D environment. The research was partially done with institutional support RVO: 67985807 and partially supported by the grant of the Czech Science Foundation AppNeCo No. GA22-02067S.

Author information

Authors and Affiliations

  1. Playtika Ltd., 1003, Lausanne, Switzerland

    Jérémie Cabessa, Hugo Hernault, Yves Lamonato, Mathieu Rochat & Yariv Z. Levy

  2. Laboratory DAVID, UVSQ – Université Paris-Saclay, 78000, Versailles, France

    Jérémie Cabessa

  3. Institute of Computer Science of the Czech Academy of Sciences, 8207, Prague 8, Czech Republic

    Jérémie Cabessa

  4. Mathematics Section (SMA), EPFL, 1015, Lausanne, Switzerland

    Mathieu Rochat

Authors
  1. Jérémie Cabessa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hugo Hernault
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yves Lamonato
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mathieu Rochat
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yariv Z. Levy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jérémie Cabessa .

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Indore, Indore, India

    Mohammad Tanveer

  2. Indian Institute of Information Technology - Allahabad, Prayagraj, India

    Sonali Agarwal

  3. Kobe University, Kobe, Japan

    Seiichi Ozawa

  4. Indian Institute of Technology Patna, Patna, India

    Asif Ekbal

  5. University of Innsbruck, Innsbruck, Austria

    Adam Jatowt

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cabessa, J., Hernault, H., Lamonato, Y., Rochat, M., Levy, Y.Z. (2023). The EsnTorch Library: Efficient Implementation of Transformer-Based Echo State Networks. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Communications in Computer and Information Science, vol 1794. Springer, Singapore. https://doi.org/10.1007/978-981-99-1648-1_20

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-1648-1_20

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-1647-4

  • Online ISBN: 978-981-99-1648-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation