Elsevier

Neurocomputing

Volume 429, 14 March 2021, Pages 132-140
Neurocomputing

Joint entity and relation extraction model based on rich semantics

https://doi.org/10.1016/j.neucom.2020.12.037Get rights and content

Highlights

  • A novel end-to-end method based on rich semantics is proposed.

  • The proposed method utilized the associated information between entities and relations.

  • The proposed method does not need to introduce external complicated features.

  • The proposed method achieves state-of-the-art performance on both NYT10 and NYT11 datasets.

Abstract

Extracting entities and relations from unstructured texts has become an important task in the natural language processing (NLP), especially knowledge graphs (KG). However, relation classification (RC) and named entity recognition (NER) tasks are usually considered separately, which lost a lot of associated contextual information. Therefore, a novel end-to-end method based on the attention mechanism integrating convolutional and recurrent neural networks is proposed for joint entity and relation extraction, which can obtain rich semantics and takes full advantage of the associated information between entities and relations without introducing external complicated features. The convolutional operation is employed to obtain character-level and word-level embeddings which are transferred to the multi-head attention mechanism. Then the multi-head attention mechanism can encode contextual semantics and embeddings to obtain efficient semantic representation. Moreover, the rich semantics are encoded to obtain final tag sequence based on recurrent neural networks. Finally, the experiments are performed on NYT10 and NYT11 benchmarks to demonstrate the proposed method. Compared with the current pipelined and joint approaches, the experimental results indicate that the proposed method can obtain state-of-the-art performance in terms of the standard F1-score.

Introduction

Joint extraction of information extraction (IE) [1] can extract entities and relations at the same time. With the development of the joint learning [2], the joint models are learned by the underlying shared representation. Owing to the development of deep neural networks, the tasks of name entity recognition (NER) [3], [4] and relation classification (RC) have successively obtained state-of-the-art results [5], [6], which play an significant role in the construction of knowledge graphs. However, the tasks of the NER and the RC in the raw text are treated as separated pipelined tasks. Although the separated process can deal with tasks easily to train and verify each task module flexibly, the correlation of both tasks are rarely considered. In addition, since the RC is followed by the NER, and errors of the NER is propagated to the RC, resulting in greater errors. Therefore, joint learning model can learn more contextual information and avoid cascading errors.

Recently, more and more joint learning models are proposed to jointly extract entities and relations. Zheng et al. proposed a joint extraction model based on the hybrid neural network [7]. The NER and the RC were regarded as two sub-tasks that shared bidirectional long short-term memory (LSTM) (Bi-LSTM) encoder to encode the representation of contextual information. Feng et al. employed the Bi-LSTM to extract entity information and tree-LSTM for representing relation mentions, respectively [8]. The information of entity recognition was delivered to relation extraction and the Q-learning algorithm was employed to make decision. Although these methods have made some breakthroughs, extracting entities and relations jointly by sharing the representation produced by the NER sub-task and the RC sub-task causes losing associated information between entities and relations.

A novel end-to-end method based on the attention mechanism integrating convolutional and recurrent neural networks is proposed for joint entity and relation extraction, which can obtain rich semantics and takes full advantage of the associated information between entities and relations without introducing external complicated features. In the lexical layer, the word-level convolutional neural network (CNN) layer and the character-level CNN layer are applied to encode features for each word. Then the multi-head attention mechanism is employed to encode multiple semantic spaces to capture long-dependency in the contextual layer. Moreover, a vector representation of the word with the underlying contextual information is obtained by concatenating outputs of the lexical layer and the contextual layer. Meanwhile, the encoder-decoder structure is effective for the tasks with long sequences [9], [10]. In the encoding layer, the Bi-LSTM is used to encode the output embedding of the concatenate layer, which learns context-dependent information. The role of the last layer is decoding, and a variant of the LSTM is applied to obtain the tags of token sentences. Experiments are performed on NYT10 and NYT11 benchmarks [11] to verify the performance of the proposed method. Compared with current pipelined and joint approaches, the experimental results show that the proposed method can obtain state-of-the-art performance in terms of the standard F1-score.

In the following, the current research results in joint entity and relation extraction are shown in Section 2 and the details of the proposed method are illustrated in Section 3. In Section 4, we display more experimental details and the extensive evaluation results compared with other baselines. Finally, Section 5 draws a conclusion.

Section snippets

Related work

The task in this paper is to extract each of entities and relations from unstructured texts at the same time. Relation extraction and entity recognition are significant tasks in constructing a knowledge base [12], [13], which are powerful support for the downstream tasks of natural language processing (NLP) [14], [15]. There are currently two solutions with pipeline and joint learning to extract entity and relation jointly [16].

For pipelined approaches, entity recognition is performed first,

Methods

The proposed method is a structure with five layers. The first layer is the lexical layer to obtain underlying word embedding and the second layer is the contextual layer to get rich representation in different semantic subspaces. Then, the third layer is the concatenate layer to concatenate the outputs of the lexical layer and the contextual layer. Finally, the encoding layer obtains more contextual information by the Bi-LSTM and the decoding layer gets the tags of sentences after decoding the

Experiments

A series of comparison experiments are designed based on the NYT10 and the NYT11 datasets to verify the effectiveness of the proposed method, as well as an ablation experiment.

Conclusion

A novel end-to-end method based on rich semantics for joint entity and relation extraction is proposed. The influence of rich semantic representation on the joint extraction in multiple semantic space is explored. The semantic representation information from different modules is concatenated, so that the word vectors can represent more comprehensive and rich contextual information of both entities and relations, which can achieve better extraction results in the decoding layer for joint

CRediT authorship contribution statement

Zhiqiang Geng: Funding acquisition, Investigation, Project administration, Resources. Yanhui Zhang: Methodology, Resources, Software, Validation, Visualization, Writing - original draft. Yongming Han: Conceptualization, Data curation, Formal analysis, Writing - review & editing, Methodology.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was partly funded by National Natural Science Foundation of China (61673046), the Fundamental Research Funds for the Central Universities (XK1802-4) and Science and Technology Major Project of Guizhou Province (Guizhou Branch [2018]3002).

Zhiqiang Geng received the B.Sc. and M.Sc. degrees from Zhengzhou University, China, in 1997 and 2002, respectively, and the Ph.D. degree in College of Information Science & Technology from Beijing University of Chemical Technology, China, in 2005. Now he is a professor of College of Information Science & Technology from Beijing University of Chemical Technology. His research interests include neural networks, intelligent computing, data mining, knowledge management and process modeling. He has

References (49)

  • Zhou P, Shi W, Tian J, Qi ZY, Li BC, Hao HW, Xu B. Attention-Based Bidirectional Long Short-Term Memory Networks for...
  • Santos N, Xiang B, Zhou BW. Classifying Relations by Ranking with Convolutional Neural Networks. arXiv...
  • Y. Feng et al.

    Joint extraction of entities and relations using reinforcement learning and deep learning

    Comput. Intel. Neurosci.

    (2017)
  • Su S, Jia NN, Cheng X, Zhu SG, Li RP. Exploring Encoder-Decoder Model for Distant Supervised Relation Extraction....
  • K. Cho et al.

    Describing multimedia content using attention-based encoder-decoder networks

    IEEE Trans. Multimedia

    (2015)
  • Takanobu R, Zhang TY, Liu JX, Huang ML. A Hierarchical Framework for Relation Extraction with Reinforcement Learning....
  • M. Nickel et al.

    A review of relational machine learning for knowledge graphs

    Proc. IEEE

    (2016)
  • Y.K. Lin et al.

    Learning entity and relation embeddings for knowledge graph completion

    Proc. Twenty-Ninth AAAI Conf. Artif. Intel.

    (2015)
  • Zheng SC, Wang F, Bao HY, Hao YX, Zhou P, Xu B. Joint Extraction of Entities and Relations Based on a Novel Tagging...
  • Yang BS, Cardie C. Joint Inference for Fine-grained Opinion Extraction. Proceedings of the 51st Annual Meeting of the...
  • Singh S, Riedel S, Martin B, Zheng JP, McCallum A. Joint Inference of Entities, Relations, and Coreference. Proceedings...
  • Choi YJ, Breck E, Cardie C. Joint extraction of entities and relations for opinion recognition. Proceedings of the 2006...
  • Miwa M, Sasaki Y. Modeling Joint Entity and Relation Extraction with Table Representation. Proceedings of the 2014...
  • Gupta P, Schtze H, Andrassy B. Table Filling Multi-Task Recurrent Neural Network for Joint Entity and Relation...
  • Cited by (67)

    • Industry-sensitive language modeling for business

      2024, European Journal of Operational Research
    View all citing articles on Scopus

    Zhiqiang Geng received the B.Sc. and M.Sc. degrees from Zhengzhou University, China, in 1997 and 2002, respectively, and the Ph.D. degree in College of Information Science & Technology from Beijing University of Chemical Technology, China, in 2005. Now he is a professor of College of Information Science & Technology from Beijing University of Chemical Technology. His research interests include neural networks, intelligent computing, data mining, knowledge management and process modeling. He has produced over 100 research papers in his fields.

    Yanhui Zhang received the B.Sc. degree in College of Information Science and Technology from Beijing University of Chemical Technology, China, in 2018. He is currently working towards the M.Sc. degree from Beijing University of Chemical Technology, China. His research interests include artificial neural networks and knowledge graph.

    Yongming Han received the B.Sc. and Ph.D. degrees from Beijing University of Chemical Technology in 2009 and 2014, respectively. Now he is a associate professor of College of Information Science & Technology from Beijing University of Chemical Technology. His research interests include knowledge graph analysis, neural networks, intelligent computing, data mining and optimization.

    View full text