融合自注意力的Bert-BiGRU-CRF的文本因果关系抽取

doi:10.3969/j.issn.1006-2475.2025.07.016

摘要/Abstract

摘要： 摘要：针对自然语言文本因果关系抽取中存在的标记方案无法处理重叠关系以及长距离依赖性的问题，本文引入tag2triplet算法来处理同一句子中的多个因果三元组和嵌入式因果关系的因果三元组，并将因果性标记方案与深度学习架构相结合用以最小化特征工程，同时有效地对因果关系建模。此外，本文将自注意力机制融合到Bert-BiGRU-CRF模型中以学习因果关系之间的长距离依赖性，允许信息在网络中自由流动，从而更准确地提取因果关系。为了验证该方法的有效性，将模型与目前广泛使用的BiLSTM-softmax模型、BiLSTM-CRF模型和Flair+CLSTM-BiLSTM-CRF模型在SemEval 2010 task8数据集上进行对比实验，结果表明，本文模型的F1评价指标分数更高，达到了83.44%。

关键词: 关键词：因果关系抽取, tag2triplet算法, Bert-BiGRU-CRF, 自注意力

Abstract: Abstract: To address the issues of overlapping relations and long-distance dependencies in causal relation extraction from natural language texts， this paper introduces the tag2triplet algorithm to handle multiple causal triplets within the same sentence and embedded causality. It combines causal labeling schemes with deep learning architectures to minimize feature engineering while effectively modeling causal relationships. Additionally， the paper integrates self-attention mechanisms into the Bert-BiGRU-CRF model to capture long-distance dependencies between causal relations， allowing information to flow freely within the network and thereby more accurately extracting causal relationships. To validate the effectiveness of this approach， the model is compared with the currently widely used BiLSTM-softmax model， BiLSTM-CRF model， and Flair + CLSTM-BiLSTM-CRF model through experiments on the SemEval 2010 task8 dataset. The results demonstrate that the proposed model achieves a higher F1 score of 83.44%.

Key words: Key words: causal relationship extraction； tag2triplet algorithm； Bert-BiGRU-CRF； self-attention ,

中图分类号:

中图分类号：TP398.1

高宁波, 张晓滨. 融合自注意力的Bert-BiGRU-CRF的文本因果关系抽取[J]. 计算机与现代化, 2025, 0(07): 112-118.

GAO Ningbo, ZHANG Xiaobin. Bert-BiGRU-CRF with Self-attention Fusion for Text Causal Relationship Extraction[J]. Computer and Modernization, 2025, 0(07): 112-118.

参考文献

［1］ KHOO C S G， KORNFILT J， ODDY R N， et al. Automatic extraction of cause-effect information from newspaper text without knowledge-based inferencing［J］. Literary and Linguistic Computing， 1998，13（4）:177-186.
［2］ KHOO C S G， MYAENG S H， ODDY R N. Using cause-effect relations in text to improve information retrieval precision［J］. Information Processing & Management， 2001，37（1）:119-145.
［3］ SILVERSTEIN C， BRIN S， MOTWANI R， et al. Scalable techniques for mining causal structures［J］. Data Mining and Knowledge Discovery， 2000，4:163-192.
［4］ RADINSKY K， DAVIDOVICH S， MARKOVITCH S. Learning causality for news events prediction［C］// Proceedings of the 21st International Conference on World Wide Web. ACM， 2012:909-918.
［5］ GIRJU R. Automatic detection of causal relations for question answering［C］// Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering. ACL， 2003:76-83.
［6］ CHANG D S， CHOI K S. Incremental cue phrase learning and bootstrapping method for causality extraction using cue phrase and word pair probabilities［J］. Information Processing & Management， 2006，42（3）:662-678.
［7］ SOBRINO A， PUENTE C， OLIVAS J A. Extracting answers from causal mechanisms in a medical document［J］. Neurocomputing， 2014，135:53-60.
［8］ RIAZ M， GIRJU R. Another look at causality: Discovering scenario-specific contingency relationships with no supervision［C］// 2010 IEEE 4th International Conference on Semantic Computing. IEEE， 2010:361-368.
［9］ HASHIMOTO C， TORISAWA K， KLOETZER J， et al. Toward future scenario generation: Extracting event causality exploiting semantic relation， context， and association features［C］// Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. ACL， 2014:987-997.
［10］ YANG J， HAN S C， POON J. A survey on extraction of causal relations from natural language text［J］. Knowledge and Information Systems， 2022，64（5）:1161-1186.
［11］ XU J H， ZUO W L， LIANG S N， et al. A review of dataset and labeling methods for causality extraction［C］// Proceedings of the 28th International Conference on Computational Linguistics. ACL， 2020:1519-1531.
［12］刘辉，江千军，桂前进，等. 实体关系抽取技术研究进展综述［J］. 计算机应用研究， 2020，37（增2）:1-5.
［13］王朱君，王石，李雪晴，等. 基于深度学习的事件因果关系抽取综述［J］. 计算机应用， 2021，41（5）:1247-1255.
［14］高强，游宏梁.事件抽取技术研究综述［J］. 情报理论与实践， 2013，36（4）:114-117.
［15］ DE SILVA T N， ZHIBO X， RUI Z， et al. Causal relation identification using convolutional neural networks and knowledge based features［J］. International Journal of Computer and Systems Engineering， 2017，11（6）: 696-701.
［16］ LI P F， MAO K Z. Knowledge-oriented convolutional neural network for causal relation extraction from natural language texts［J］. Expert Systems with Applications， 2019，115:512-523.
［17］ JIN G G， ZHOU J S， QU W G， et al. Exploiting rich event representation to improve event causality recognition［J］. Intelligent Automation and Soft Computing， 2021，30（1）:161-173.
［18］ CAO P F， ZUO X Y， CHEN Y B， et al. Knowledge-enriched event causality identification via latent structure induction networks［C］// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL， 2021:4862-4872.
［19］ ZHENG S， WANG F， BAO H， et al. Joint extraction of entities and relations based on a novel tagging scheme［J］. arXiv preprint arXiv:1706.05075， 2017.
［20］ XU J H， ZUO W L， LIANG S N， et al. Causal relation extraction based on graph attention networks［J］. Journal of Computer Research and Development， 2020，57（1）:159-174.
［21］ VELIČKOVIĆ P， CUCURULL G， CASANOVA A， et al. Graph attention networks［J］. arXiv preprint arXiv:1710.10903， 2017.
［22］ ZHENG S C， WANG F， BAO H Y， et al. Joint extraction of entities and relations based on a novel tagging scheme［C］// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. ACL， 2017:1227-1236.
［23］ KABIR M A， ALMULHIM A J， LUO X， et al. Informative causality extraction from medical literature via dependency-tree-based patterns［J］. Journal of Healthcare Informatics Research， 2022，6（3）:295-316.
［24］ LI Z N， LI Q， ZOU X T， et al. Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings［J］. Neurocomputing， 2021，423:207-219.
［25］陈德光，马金林，马自萍，等. 自然语言处理预训练技术综述［J］. 计算机科学与探索， 2021，15（8）:1359-1389.
［26］ HOCHREITER S， SCHMIDHUBER J. Long short-term memory［J］. Neural Computation， 1997，9（8）:1735-1780.
［27］ CHUNG J， GULCEHRE C， CHO K H， et al. Empirical evaluation of gated recurrent neural networks on sequence modeling［J］. arXiv preprint arXiv:1412.3555， 2014.
［28］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM， 2017:6000-6010.
［29］ LAFFERTY J D， MCCALLUM A， PEREIRA F C N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data［C］// Proceedings of the 18th International Conference on Machine Learning. ACM， 2001:282-289.
［30］ SORGENTE A， VETTIGLI G， MELE F. Automatic extraction of cause-effect relations in natural language text［J］. CEUR Workshop Proceedings， 2013，1109:37-48.
［31］ LUO Z Y， SHA Y C， ZHU K Q， et al. Commonsense causal reasoning between short texts［C］// 15th International Conference on the Principles of Knowledge Representation and Reasoning. ACM， 2016:421-430.
［32］ WANG P L， QIAN Y， SOONG F K， et al. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network［J］. arXiv preprint arXiv:1510.06168， 2015.
［33］ HUANG Z H， XU W， YU K. Bidirectional LSTM-CRF models for sequence tagging［J］. arXiv preprint arXiv:1508.01991， 2015.
［34］ LAMPLE G， BALLESTEROS M， SUBRAMANIAN S， et al. Neural architectures for named entity recognition［C］// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL， 2016:260-270.
［35］ MA X， HOVY E. End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF［C］// 54th Annual Meeting of the Association for Computational Linguistics. ACL， 2016: 1064-1074.
［36］ DEVLIN J， CHANG M W， LEE K， et al. Bert: Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL， 2018:4171-4186.
［37］ AKBIK A， BLYTHE D， VOLLGRAF R. Contextual string embeddings for sequence labeling［C］// Proceedings of the 27th International Conference on Computational Linguistics. ACL， 2018:1638-1649.
［38］ CAI G P， SU X H， WU T. Causality extraction of fused character features with BiGRU-Attention-CRF[J]. Cognitive Robotics， 2022. DOI: 10.2139/ssrn.4211811.