Joint entity and relation extraction model based on rich semantics
Introduction
Joint extraction of information extraction (IE) [1] can extract entities and relations at the same time. With the development of the joint learning [2], the joint models are learned by the underlying shared representation. Owing to the development of deep neural networks, the tasks of name entity recognition (NER) [3], [4] and relation classification (RC) have successively obtained state-of-the-art results [5], [6], which play an significant role in the construction of knowledge graphs. However, the tasks of the NER and the RC in the raw text are treated as separated pipelined tasks. Although the separated process can deal with tasks easily to train and verify each task module flexibly, the correlation of both tasks are rarely considered. In addition, since the RC is followed by the NER, and errors of the NER is propagated to the RC, resulting in greater errors. Therefore, joint learning model can learn more contextual information and avoid cascading errors.
Recently, more and more joint learning models are proposed to jointly extract entities and relations. Zheng et al. proposed a joint extraction model based on the hybrid neural network [7]. The NER and the RC were regarded as two sub-tasks that shared bidirectional long short-term memory (LSTM) (Bi-LSTM) encoder to encode the representation of contextual information. Feng et al. employed the Bi-LSTM to extract entity information and tree-LSTM for representing relation mentions, respectively [8]. The information of entity recognition was delivered to relation extraction and the Q-learning algorithm was employed to make decision. Although these methods have made some breakthroughs, extracting entities and relations jointly by sharing the representation produced by the NER sub-task and the RC sub-task causes losing associated information between entities and relations.
A novel end-to-end method based on the attention mechanism integrating convolutional and recurrent neural networks is proposed for joint entity and relation extraction, which can obtain rich semantics and takes full advantage of the associated information between entities and relations without introducing external complicated features. In the lexical layer, the word-level convolutional neural network (CNN) layer and the character-level CNN layer are applied to encode features for each word. Then the multi-head attention mechanism is employed to encode multiple semantic spaces to capture long-dependency in the contextual layer. Moreover, a vector representation of the word with the underlying contextual information is obtained by concatenating outputs of the lexical layer and the contextual layer. Meanwhile, the encoder-decoder structure is effective for the tasks with long sequences [9], [10]. In the encoding layer, the Bi-LSTM is used to encode the output embedding of the concatenate layer, which learns context-dependent information. The role of the last layer is decoding, and a variant of the LSTM is applied to obtain the tags of token sentences. Experiments are performed on NYT10 and NYT11 benchmarks [11] to verify the performance of the proposed method. Compared with current pipelined and joint approaches, the experimental results show that the proposed method can obtain state-of-the-art performance in terms of the standard F1-score.
In the following, the current research results in joint entity and relation extraction are shown in Section 2 and the details of the proposed method are illustrated in Section 3. In Section 4, we display more experimental details and the extensive evaluation results compared with other baselines. Finally, Section 5 draws a conclusion.
Section snippets
Related work
The task in this paper is to extract each of entities and relations from unstructured texts at the same time. Relation extraction and entity recognition are significant tasks in constructing a knowledge base [12], [13], which are powerful support for the downstream tasks of natural language processing (NLP) [14], [15]. There are currently two solutions with pipeline and joint learning to extract entity and relation jointly [16].
For pipelined approaches, entity recognition is performed first,
Methods
The proposed method is a structure with five layers. The first layer is the lexical layer to obtain underlying word embedding and the second layer is the contextual layer to get rich representation in different semantic subspaces. Then, the third layer is the concatenate layer to concatenate the outputs of the lexical layer and the contextual layer. Finally, the encoding layer obtains more contextual information by the Bi-LSTM and the decoding layer gets the tags of sentences after decoding the
Experiments
A series of comparison experiments are designed based on the NYT10 and the NYT11 datasets to verify the effectiveness of the proposed method, as well as an ablation experiment.
Conclusion
A novel end-to-end method based on rich semantics for joint entity and relation extraction is proposed. The influence of rich semantic representation on the joint extraction in multiple semantic space is explored. The semantic representation information from different modules is concatenated, so that the word vectors can represent more comprehensive and rich contextual information of both entities and relations, which can achieve better extraction results in the decoding layer for joint
CRediT authorship contribution statement
Zhiqiang Geng: Funding acquisition, Investigation, Project administration, Resources. Yanhui Zhang: Methodology, Resources, Software, Validation, Visualization, Writing - original draft. Yongming Han: Conceptualization, Data curation, Formal analysis, Writing - review & editing, Methodology.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This research was partly funded by National Natural Science Foundation of China (61673046), the Fundamental Research Funds for the Central Universities (XK1802-4) and Science and Technology Major Project of Guizhou Province (Guizhou Branch [2018]3002).
Zhiqiang Geng received the B.Sc. and M.Sc. degrees from Zhengzhou University, China, in 1997 and 2002, respectively, and the Ph.D. degree in College of Information Science & Technology from Beijing University of Chemical Technology, China, in 2005. Now he is a professor of College of Information Science & Technology from Beijing University of Chemical Technology. His research interests include neural networks, intelligent computing, data mining, knowledge management and process modeling. He has
References (49)
- et al.
Bilinear joint learning of word and entity embeddings for Entity Linking
Neurocomputing
(2018) - et al.
SANE 2.0: system for fine grained named entity typing on textual data
Eng. Appl. Artif. Intel.
(2019) - et al.
Joint entity and relation extraction based on a hybrid neural network
Neurocomputing
(2017) - et al.
An asymmetric knowledge representation learning in manifold space
Inform. Sci.
(2020) - et al.
Knowledge graph based natural language generation with adapted pointer-generator networks
Neurocomputing
(2020) - et al.
Energy optimization and prediction modeling of petrochemical industries: an improved convolutional neural network based on cross-feature
Energy
(2020) - et al.
Novel leakage detection and water loss management of urban water supply network using multiscale neural networks
J. Cleaner Prod.
(2021) - et al.
Semantic relation extraction using sequential and tree-structured LSTM with attention
Inform. Sci.
(2020) - et al.
A survey of web information extraction systems
IEEE Trans. Knowl. Data Eng.
(2006) - et al.
arabic named entity recognition: a feature-driven study
IEEE Trans. Audio Speech Lang. Process.
(2009)
Joint extraction of entities and relations using reinforcement learning and deep learning
Comput. Intel. Neurosci.
Describing multimedia content using attention-based encoder-decoder networks
IEEE Trans. Multimedia
A review of relational machine learning for knowledge graphs
Proc. IEEE
Learning entity and relation embeddings for knowledge graph completion
Proc. Twenty-Ninth AAAI Conf. Artif. Intel.
Cited by (67)
Industry-sensitive language modeling for business
2024, European Journal of Operational ResearchA contrastive learning framework for safety information extraction in construction
2023, Advanced Engineering InformaticsA distributed joint extraction framework for sedimentological entities and relations with federated learning
2023, Expert Systems with ApplicationsA new method to extract n-Ary relation instances from scientific documents
2022, Expert Systems with Applications
Zhiqiang Geng received the B.Sc. and M.Sc. degrees from Zhengzhou University, China, in 1997 and 2002, respectively, and the Ph.D. degree in College of Information Science & Technology from Beijing University of Chemical Technology, China, in 2005. Now he is a professor of College of Information Science & Technology from Beijing University of Chemical Technology. His research interests include neural networks, intelligent computing, data mining, knowledge management and process modeling. He has produced over 100 research papers in his fields.
Yanhui Zhang received the B.Sc. degree in College of Information Science and Technology from Beijing University of Chemical Technology, China, in 2018. He is currently working towards the M.Sc. degree from Beijing University of Chemical Technology, China. His research interests include artificial neural networks and knowledge graph.
Yongming Han received the B.Sc. and Ph.D. degrees from Beijing University of Chemical Technology in 2009 and 2014, respectively. Now he is a associate professor of College of Information Science & Technology from Beijing University of Chemical Technology. His research interests include knowledge graph analysis, neural networks, intelligent computing, data mining and optimization.