Bert-BiGRU-CRF with Self-attention Fusion for Text Causal Relationship Extraction

doi:10.3969/j.issn.1006-2475.2025.07.016

Abstract

Abstract: Abstract: To address the issues of overlapping relations and long-distance dependencies in causal relation extraction from natural language texts， this paper introduces the tag2triplet algorithm to handle multiple causal triplets within the same sentence and embedded causality. It combines causal labeling schemes with deep learning architectures to minimize feature engineering while effectively modeling causal relationships. Additionally， the paper integrates self-attention mechanisms into the Bert-BiGRU-CRF model to capture long-distance dependencies between causal relations， allowing information to flow freely within the network and thereby more accurately extracting causal relationships. To validate the effectiveness of this approach， the model is compared with the currently widely used BiLSTM-softmax model， BiLSTM-CRF model， and Flair + CLSTM-BiLSTM-CRF model through experiments on the SemEval 2010 task8 dataset. The results demonstrate that the proposed model achieves a higher F1 score of 83.44%.

Key words: Key words: causal relationship extraction； tag2triplet algorithm； Bert-BiGRU-CRF； self-attention ,

CLC Number:

中图分类号：TP398.1

GAO Ningbo, ZHANG Xiaobin. Bert-BiGRU-CRF with Self-attention Fusion for Text Causal Relationship Extraction[J]. Computer and Modernization, 2025, 0(07): 112-118.

References

［1］ KHOO C S G， KORNFILT J， ODDY R N， et al. Automatic extraction of cause-effect information from newspaper text without knowledge-based inferencing［J］. Literary and Linguistic Computing， 1998，13（4）:177-186.
［2］ KHOO C S G， MYAENG S H， ODDY R N. Using cause-effect relations in text to improve information retrieval precision［J］. Information Processing & Management， 2001，37（1）:119-145.
［3］ SILVERSTEIN C， BRIN S， MOTWANI R， et al. Scalable techniques for mining causal structures［J］. Data Mining and Knowledge Discovery， 2000，4:163-192.
［4］ RADINSKY K， DAVIDOVICH S， MARKOVITCH S. Learning causality for news events prediction［C］// Proceedings of the 21st International Conference on World Wide Web. ACM， 2012:909-918.
［5］ GIRJU R. Automatic detection of causal relations for question answering［C］// Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering. ACL， 2003:76-83.
［6］ CHANG D S， CHOI K S. Incremental cue phrase learning and bootstrapping method for causality extraction using cue phrase and word pair probabilities［J］. Information Processing & Management， 2006，42（3）:662-678.
［7］ SOBRINO A， PUENTE C， OLIVAS J A. Extracting answers from causal mechanisms in a medical document［J］. Neurocomputing， 2014，135:53-60.
［8］ RIAZ M， GIRJU R. Another look at causality: Discovering scenario-specific contingency relationships with no supervision［C］// 2010 IEEE 4th International Conference on Semantic Computing. IEEE， 2010:361-368.
［9］ HASHIMOTO C， TORISAWA K， KLOETZER J， et al. Toward future scenario generation: Extracting event causality exploiting semantic relation， context， and association features［C］// Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. ACL， 2014:987-997.
［10］ YANG J， HAN S C， POON J. A survey on extraction of causal relations from natural language text［J］. Knowledge and Information Systems， 2022，64（5）:1161-1186.
［11］ XU J H， ZUO W L， LIANG S N， et al. A review of dataset and labeling methods for causality extraction［C］// Proceedings of the 28th International Conference on Computational Linguistics. ACL， 2020:1519-1531.
［12］刘辉，江千军，桂前进，等. 实体关系抽取技术研究进展综述［J］. 计算机应用研究， 2020，37（增2）:1-5.
［13］王朱君，王石，李雪晴，等. 基于深度学习的事件因果关系抽取综述［J］. 计算机应用， 2021，41（5）:1247-1255.
［14］高强，游宏梁.事件抽取技术研究综述［J］. 情报理论与实践， 2013，36（4）:114-117.
［15］ DE SILVA T N， ZHIBO X， RUI Z， et al. Causal relation identification using convolutional neural networks and knowledge based features［J］. International Journal of Computer and Systems Engineering， 2017，11（6）: 696-701.
［16］ LI P F， MAO K Z. Knowledge-oriented convolutional neural network for causal relation extraction from natural language texts［J］. Expert Systems with Applications， 2019，115:512-523.
［17］ JIN G G， ZHOU J S， QU W G， et al. Exploiting rich event representation to improve event causality recognition［J］. Intelligent Automation and Soft Computing， 2021，30（1）:161-173.
［18］ CAO P F， ZUO X Y， CHEN Y B， et al. Knowledge-enriched event causality identification via latent structure induction networks［C］// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL， 2021:4862-4872.
［19］ ZHENG S， WANG F， BAO H， et al. Joint extraction of entities and relations based on a novel tagging scheme［J］. arXiv preprint arXiv:1706.05075， 2017.
［20］ XU J H， ZUO W L， LIANG S N， et al. Causal relation extraction based on graph attention networks［J］. Journal of Computer Research and Development， 2020，57（1）:159-174.
［21］ VELIČKOVIĆ P， CUCURULL G， CASANOVA A， et al. Graph attention networks［J］. arXiv preprint arXiv:1710.10903， 2017.
［22］ ZHENG S C， WANG F， BAO H Y， et al. Joint extraction of entities and relations based on a novel tagging scheme［C］// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. ACL， 2017:1227-1236.
［23］ KABIR M A， ALMULHIM A J， LUO X， et al. Informative causality extraction from medical literature via dependency-tree-based patterns［J］. Journal of Healthcare Informatics Research， 2022，6（3）:295-316.
［24］ LI Z N， LI Q， ZOU X T， et al. Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings［J］. Neurocomputing， 2021，423:207-219.
［25］陈德光，马金林，马自萍，等. 自然语言处理预训练技术综述［J］. 计算机科学与探索， 2021，15（8）:1359-1389.
［26］ HOCHREITER S， SCHMIDHUBER J. Long short-term memory［J］. Neural Computation， 1997，9（8）:1735-1780.
［27］ CHUNG J， GULCEHRE C， CHO K H， et al. Empirical evaluation of gated recurrent neural networks on sequence modeling［J］. arXiv preprint arXiv:1412.3555， 2014.
［28］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM， 2017:6000-6010.
［29］ LAFFERTY J D， MCCALLUM A， PEREIRA F C N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data［C］// Proceedings of the 18th International Conference on Machine Learning. ACM， 2001:282-289.
［30］ SORGENTE A， VETTIGLI G， MELE F. Automatic extraction of cause-effect relations in natural language text［J］. CEUR Workshop Proceedings， 2013，1109:37-48.
［31］ LUO Z Y， SHA Y C， ZHU K Q， et al. Commonsense causal reasoning between short texts［C］// 15th International Conference on the Principles of Knowledge Representation and Reasoning. ACM， 2016:421-430.
［32］ WANG P L， QIAN Y， SOONG F K， et al. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network［J］. arXiv preprint arXiv:1510.06168， 2015.
［33］ HUANG Z H， XU W， YU K. Bidirectional LSTM-CRF models for sequence tagging［J］. arXiv preprint arXiv:1508.01991， 2015.
［34］ LAMPLE G， BALLESTEROS M， SUBRAMANIAN S， et al. Neural architectures for named entity recognition［C］// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL， 2016:260-270.
［35］ MA X， HOVY E. End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF［C］// 54th Annual Meeting of the Association for Computational Linguistics. ACL， 2016: 1064-1074.
［36］ DEVLIN J， CHANG M W， LEE K， et al. Bert: Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL， 2018:4171-4186.
［37］ AKBIK A， BLYTHE D， VOLLGRAF R. Contextual string embeddings for sequence labeling［C］// Proceedings of the 27th International Conference on Computational Linguistics. ACL， 2018:1638-1649.
［38］ CAI G P， SU X H， WU T. Causality extraction of fused character features with BiGRU-Attention-CRF[J]. Cognitive Robotics， 2022. DOI: 10.2139/ssrn.4211811.

[1]	ZHANG Shanlu1, ZHANG Wei2. Face Anti-spoofing Based on Domain Synthesis and Contrastive Learning [J]. Computer and Modernization, 2025, 0(07): 1-8.
[2]	JIANG Zhiwei1, FU Xiaojin1, CHEN Wenbin1, JIANG Yichen2. Insulator Defect Detection by UAV Based on Lightweight YOLOv8 [J]. Computer and Modernization, 2025, 0(07): 9-14.
[3]	WANG Tao1, WU Qian1, 2. ADHD Classification Based on ConvNeXt and Attention [J]. Computer and Modernization, 2025, 0(07): 15-20.
[4]	SUN Kaijie, HU Jili. Application of Multimodal Large Language Models in Diagnosis of Pigmented Skin Lesions [J]. Computer and Modernization, 2025, 0(07): 21-27.
[5]	HUA Man, LIU Xiaoliang. Android Malicious Application Detection Based on RA-CNN and Residual Network [J]. Computer and Modernization, 2025, 0(07): 28-32.
[6]	HE Lili1, 2, ZHANG Chenglin1, 2, ZHANG Lei1, 2 , CAO Mingzeng1, 2. Double Privacy Protection Algorithm for User Location Based on Multi-anonymizer Architecture [J]. Computer and Modernization, 2025, 0(07): 33-42.
[7]	KUANG Ye1, ZHOU Mo2, LIU Ceyue1. A Survey of Network Outage Detection [J]. Computer and Modernization, 2025, 0(07): 43-54.
[8]	CAI Bohan, LIU Jun. Long-and Short-Term Air Pollutant Concentration Forecasting Based on Optimized Transformer [J]. Computer and Modernization, 2025, 0(07): 55-62.
[9]	LI Zihe, WANG Yiding. 3D Human Motion Similarity Estimation from Different Perspectives [J]. Computer and Modernization, 2025, 0(07): 63-68.
[10]	JIAO Leyan, ZHU Xinjuan. Speech Cloning Method Based on Self-attention Mechanism Speaker Encoder And SA-Decoder [J]. Computer and Modernization, 2025, 0(07): 69-76.
[11]	RAO Hongyu1, CHEN Xin2, CHEN Sheng3, XIA Tian3, SHEN Li4, LUY Guangqiang5. Power Load Forecasting of Bi-LSTM Based on Improving Whale Algorithm [J]. Computer and Modernization, 2025, 0(07): 77-82.
[12]	ZHAI Junlong, GU Lin. Multi-modal Emotion Recognition Based on Text Guidance [J]. Computer and Modernization, 2025, 0(07): 83-89.
[13]	DENG Chao1, 2, YANG Fengkun1, 2, LI Jun3, CHEN Liangliang1, 2, GUO Zhichong4. Complex Fault Diagnosis Method of Charging Facilities Based on Bi-LSTM Residual Network [J]. Computer and Modernization, 2025, 0(07): 90-96.
[14]	WEI Qingsong1, WANG Xiaojun2, WEI Yuan1, ZENG Shangqi1, FANG Yuan1. HGAT: Multivariate Time Series Anomaly Detection Based on Hybrid Graph Attention Network [J]. Computer and Modernization, 2025, 0(07): 97-105.
[15]	WU Dong, FAN Yongsheng, SAN Binbin. Lightweight Flame Detection Algorithm Based on Improved RT-DETR [J]. Computer and Modernization, 2025, 0(07): 106-111.