Abstract
In the last years, innovative techniques like Transfer Learning have impacted strongly in Natural Language Processing, increasing massively the state-of-the-art in several challenging tasks. In particular, the Universal Language Model Fine-Tuning (ULMFiT) algorithm has proven to have an impressive performance on several English text classification tasks. In this paper, we aim at developing an algorithm for Spanish Sentiment Analysis of short texts that is comparable to the state-of-the-art. In order to do so, we have adapted the ULMFiT algorithm to this setting. Experimental results on benchmark datasets (InterTASS 2017 and InterTASS 2018) show how this simple transfer learning approach performs well when compared to fancy deep learning techniques.
This work was funded by CONCYTEC-FONDECYT under the call E041-01 [contract number 34-2018-FONDECYT-BM-IADT-SE].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arora, S., Liang, Y., Ma, T.: A simple but tough-to-beat baseline for sentence embeddings. In: International Conference on Learning Representations (2017)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Brooke, J., Tofiloski, M., Taboada, M.: Cross-linguistic sentiment analysis: From English to Spanish. In: Proceedings of RANLP, vol. 2009, pp. 50–54 (2009)
Chiruzzo, L., Rosá, A.: RETUYT-InCo at TASS 2018: sentiment analysis in Spanish variants using neural networks and SVM. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 57–63 (2018)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018)
Fastai: Fastai, May 2019. https://github.com/fastai/fastai
Fastai: ULMFiT, May 2019. https://github.com/fastai/fastai/tree/master/courses/dl2/imdb_scripts
Garcia, M., Martinez, E., Villena, J., Garcia, J.: Tass 2015 - the evolution of the spanish opinion mining systems. Procesamiento de Lenguaje Natural 56, 33–40 (2016)
Garcia-Cumbreras, M.A., Villena-Roman, J., Martinez-Camara, E., Diaz-Galiano, M., Martin-Valdivia, T., Ureña Lopez, A.: Overview of TASS 2016. In: Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN, pp. 13–21 (2016)
Gonzalez, J.A., Hurtado, L.F., Pla, F.: ELiRF-UPV at TASS 2018: sentiment analysis in twitter based on deep learning. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 37–44 (2018)
Graves, A.: Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-24797-2. https://cds.cern.ch/record/1503877
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328–339. Association for Computational Linguistics, Melbourne, July 2018. https://www.aclweb.org/anthology/P18-1031
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1746–1751 (2014). http://aclweb.org/anthology/D/D14/D14-1181.pdf
Liu, B.: Sentiment Analysis and Opinion Mining. Morgan and Claypool Publishers, San Rafael (2012)
Martinez-Camara, E., et al.: Overview of TASS 2018: opinions, health and emotions. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 13–27 (2018)
Martinez-Camara, E., Diaz-Galiano, M., Garcia-Cumbreras, M.A., Garcia-Vega, M., Villena-Roman, J.: Overview of TASS 2017. In: Proceedings of TASS 2017: Workshop on Sentiment Analysis at SEPLN, pp. 13–21 (2017)
McGlohon, M., Glance, N., Reiter, Z.: Star quality: aggregating reviews to rank products and merchants. In: Proceedings of Fourth International Conference on Weblogs and Social Media (ICWSM) (2010)
Merity, S., Keskar, N.S., Socher, R.: Regularizing and optimizing LSTM language models. CoRR abs/1708.02182 (2017). http://arxiv.org/abs/1708.02182
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 3111–3119. Curran Associates, Inc. (2013). http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
Montañes, R., Aznar, R., del Hoyo, R.: Application of a hybrid deep learning model for sentiment analysis in Twitter. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 51–56 (2018)
Ochoa-Luna, J., Ari, D.: Deep neural network approaches for spanish sentiment analysis of short texts. In: Simari, G.R., Fermé, E., Gutiérrez Segura, F., Rodríguez Melquiades, J.A. (eds.) IBERAMIA 2018. LNCS (LNAI), vol. 11238, pp. 430–441. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03928-8_35
Palomino, D.: ULMFit implementation for TASS dataset evaluation, May 2019. https://github.com/dpalominop/ULMFit
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162
Peters, M., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana, June 2018. https://doi.org/10.18653/v1/N18-1202, https://www.aclweb.org/anthology/N18-1202
Rother, K., Rettberg, A.: ULMFiT at GermEval-2018: a deep neural language model for the classification of hate speech in German Tweets. In: Proceedings of the GermEval 2018 Workshop, pp. 113–119 (2018)
Tang, D., Wei, F., Qin, B., Yang, N., Liu, T., Zhou, M.: Sentiment embeddings with applications to sentiment analysis. IEEE Trans. Knowl. Data Eng. 28(2), 496–509 (2016)
Wu, X., Lv, S., Zang, L., Han, J., Hu, S.: Conditional BERT contextual augmentation. CoRR abs/1812.06705 (2018). http://arxiv.org/abs/1812.06705
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Palomino, D., Ochoa-Luna, J. (2019). Advanced Transfer Learning Approach for Improving Spanish Sentiment Analysis. In: Martínez-Villaseñor, L., Batyrshin, I., Marín-Hernández, A. (eds) Advances in Soft Computing. MICAI 2019. Lecture Notes in Computer Science(), vol 11835. Springer, Cham. https://doi.org/10.1007/978-3-030-33749-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-33749-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33748-3
Online ISBN: 978-3-030-33749-0
eBook Packages: Computer ScienceComputer Science (R0)