Skip to main content
SpringerLink
Log in

Text Classification with Attention Gated Graph Neural Network

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Text classification is a fundamental and important task in natural language processing. There have been many graph-based neural networks for this task with the capacity of learning complicated relational information between word nodes. However, existing approaches are potentially insufficient in capturing semantic relationships between the words. In this paper, to address the above issue, we propose a novel graph-based model where every document is represented as a text graph. Specifically, we devise an attention gated graph neural network (AGGNN) to propagate and update the semantic information of each word node from their 1-hop neighbors. Keyword nodes with discriminative semantic information are extracted via our proposed attention-based text pooling layer (TextPool), which also aggregates the document embedding. In this case, text classification is transformed into a graph classification task. Extensive experiments on four benchmark datasets demonstrate that the proposed model outperforms other previous text classification approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. http://disi.unitn.it/moschitti/corpora.htm

  2. https://www.cs.umb.edu/~smimarog/textmining/datasets/

  3. http://www.cs.cornell.edu/people/pabo/movie-review-data/

  4. http://www.nltk.org/

References

  1. Aggarwal CC, Zhai C. A survey of text classification algorithms. In: Mining Text Data. Springer; 2012. p. 163-222.

  2. Peng F, Schuurmans D. Combining naive bayes and n-gram language models for text classification. In: Proceedings of the European Conference on Information Retrieval Research; 2003. p. 335-50.

  3. Joachims T. Text categorization with support vector machines: learning with many relevant features. In: Proceedings of the European Conference on Machine Learning; 1998. p. 137-42.

  4. Kang H, Nam K, Kim S. The decomposed k-nearest neighbor algorithm for imbalanced text classification. In: Proceedings of International Conference on Future Generation Information Technology; 2012. p. 87-94.

  5. Kim Y. Convolutional neural networks for sentence classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2014. p. 1746-51.

  6. Mikolov T, Karafiát M, Burget L, Cernocký J, Khudanpur S. Recurrent neural network based language model. In: Proceedings of the Annual Conference of the International Speech Communication Association; 2010. p. 1045-8.

  7. Lai S, Xu L, Liu K, Zhao J. Recurrent convolutional neural networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2015. p. 2267-73.

  8. Wang R, Li Z, Cao J, Chen T, Wang L. Convolutional recurrent neural networks for text classification. In: Proceedings of the International Joint Conference on Neural Networks; 2019. p. 1-6.

  9. Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G. The graph neural network model. IEEE Transactions on Neural Networks. 2009;20(1):61–80.

    Article  Google Scholar 

  10. Yao L, Mao C, Luo Y. Graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2019. p. 7370-7.

  11. Wang SI, Manning CD. Baselines and bigrams: simple, good sentiment and topic classification. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics; 2012. p. 90-4.

  12. Chenthamarakshan V, Melville P, Sindhwani V, Lawrence RD. Concept labeling: building text classifiers with minimal supervision. In: Proceedings of the International Joint Conference on Artificial Intelligence; 2011. p. 1225-30.

  13. Luo Y, Uzuner Ö, Szolovits P. Bridging semantics and syntax with graph algorithms - state-of-the-art of extracting biomedical relations. Briefings Bioinform. 2017;18(1):160–78.

    Article  Google Scholar 

  14. Rousseau F, Kiagias E, Vazirgiannis M. Text categorization as a graph classification problem. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics; 2015. p. 1702-12.

  15. Skianis K, Rousseau F, Vazirgiannis M. Regularizing text categorization with clusters of words. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2016. p. 1827-37.

  16. Tai KS, Socher R, Manning CD. Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics; 2015. p. 1556-66.

  17. Liu P, Qiu X, Huang X. Recurrent neural network for text classification with multi-task learning. In: Proceedings of the International Joint Conference on Artificial Intelligence; 2016. p. 2873-9.

  18. Joulin A, Grave E, Bojanowski P, Mikolov T. Bag of tricks for efficient text classification. In: Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics; 2017. p. 427-31.

  19. Wang G, Li C, Wang W, Zhang Y, Shen D, Zhang X, et al. Joint embedding of words and labels for text classification. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics; 2018. p. 2321-31.

  20. Zhou J, Cui G, Zhang Z, Yang C, Liu Z, Sun M. Graph neural networks: a review of methods and applications. arXiv preprint arXiv:181208434. 2018.

  21. Zhang Z, Cui P, Zhu W. Deep learning on graphs: a survey. arXiv preprint arXiv:181204202. 2018.

  22. Fu X, Zhang J, Meng Z, King I. MAGNN: Metapath aggregated graph neural network for heterogeneous graph embedding. In: Proceedings of The Web Conference 2020; 2020. p. 2331–2341.

  23. Kipf TN, Welling M. Semi-supervised classification with graph convolutional networks. In: Proceedings of the International Conference on Learning Representations; 2017. .

  24. Huang L, Ma D, Li S, Zhang X, Wang H. Text level graph neural network for text classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2019. p. 3442-8.

  25. Zhang Y, Yu X, Cui Z, Wu S, Wen Z, Wang L. Every document owns its structure: inductive text classification via graph neural networks. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics; 2020. p. 334-9.

  26. Jose JM, Yilmaz E, Magalhães J, Castells P, Ferro N, Silva MJ, et al. VGCN-BERT: Augmenting BERT with graph embedding for text classification. In: Advances in Information Retrieval: 42nd European Conference on IR Research; 2020. p. 369-82.

  27. Liu X, You X, Zhang X, Wu J, Lv P. Tensor graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2020. p. 8409-16.

  28. Li C, Peng X, Peng H, Wang L, Li J. TextGTL: Graph-based transductive learning for semi-supervised text classification via structure-sensitive interpolation. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence; 2021. p. 2680-6.

  29. Blanco R, Lioma C. Graph-based term weighting for information retrieval. Inf Retr. 2012;15(1):54–92.

    Article  Google Scholar 

  30. Li Y, Tarlow D, Brockschmidt M, Zemel RS. Gated graph sequence neural networks. In: Proceedings of the International Conference on Learning Representations; 2016. .

  31. Huang J, Li Z, Li N, Liu S, Li G. AttPool: Towards hierarchical feature representation in graph convolutional networks via attention mechanism. In: Proceedings of the IEEE International Conference on Computer Vision; 2019. p. 6479-88.

  32. Li Q, Han Z, Wu XM. Deeper insights into graph convolutional networks for semi-supervised learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2018;32(1).

  33. Tang J, Qu M, Mei Q. Pte: Predictive text embedding through large-scale heterogeneous text networks. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2015. p. 1165-74.

  34. Granmo O. The tsetlin machine - a game theoretic bandit driven approach to optimal pattern recognition with propositional logic. CoRR. 2018;abs/1804.01508.

  35. Yadav RK, Jiao L, Granmo OC, Goodwin M. Enhancing interpretable clauses semantically using pretrained word representation. arXiv preprint arXiv:210406901. 2021.

  36. Shen D, Wang G, Wang W, Min MR, Su Q, Zhang Y, et al. Baseline needs more love: on simple word-embedding-based models and associated pooling mechanisms. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics; 2018. p. 440-50.

  37. Defferrard M, Bresson X, Vandergheynst P. Convolutional neural networks on graphs with fast localized spectral filtering. In: Proceedings of the Annual Conference on Neural Information Processing Systems; 2016. p. 3837-45.

  38. Zhu H, Koniusz P. Simple spectral graph convolution. In: International Conference on Learning Representations; 2020.

  39. Kingma DP, Ba J. Adam: A method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations; 2015.

  40. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics; 2010. p. 249-56.

  41. Pennington J, Socher R, Manning CD. Glove: Global vectors for word representation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2014. p. 1532-43.

Download references

Acknowledgements

This work was partially supported by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, the Natural Science Foundation of Shandong Province under Grants No. ZR2020MF131 and No. ZR2021ZD19, and the Science and Technology Program of Qingdao under Grant No. 21-1-4-ny-19-nsh. The authors thank Ke Xu for his help in the revision of this paper.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Ocean University of China, Qingdao, 266100, China

    Zhaoyang Deng, Chenxiang Sun, Guoqiang Zhong & Yuxu Mao

Authors
  1. Zhaoyang Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenxiang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guoqiang Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuxu Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guoqiang Zhong.

Ethics declarations

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, Z., Sun, C., Zhong, G. et al. Text Classification with Attention Gated Graph Neural Network. Cogn Comput 14, 1464–1473 (2022). https://doi.org/10.1007/s12559-022-10017-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-022-10017-3

Keywords

Search

Navigation