Review of Relation Extraction Based on Pre-training Language Model

Abstract

Abstract: In recent years, with the continuous innovation of deep learning technology， the application of pre-training models in natural language processing has become more and more extensive, and relation extraction is no longer purely dependent on the traditional pipeline method. The development of pre-training language models has greatly promoted the related research of relation extraction， and has surpassed traditional methods in many fields. First， this paper briefly introduces the development of relationship extraction and classic pre-training models；secondly, summarizes the current commonly used data sets and evaluation methods, and analyzes the performance of the model on each data set; finally， discusses the development challenges of relationship extraction and future research trends.

Key words: deep learning, pre-training model, relation extraction, feature extraction, natural language processing

WANG Hao-chang, LIU Ru-yi. Review of Relation Extraction Based on Pre-training Language Model[J]. Computer and Modernization, 2023, 0(01): 49-57.

References

［1］ CHINCHOR N， MARSH E. MUC-7 information extraction task definition［C］// Proceedings of the 7th Message Understanding Conference. 1998:359-367.
［2］ NIST Website. Automatic Content Extraction［EB/OL］.［2007-05-28］. http://www.nist.gov/speech/tests/ace/.
［3］ HENDRICKX I， KIM S N， KOZAREVA Z， et al. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals［C］// Proceedings of the 2009 Workshop on Semantic Evaluations: Recent Achievements and Future Directions. 2009:94-99.
［4］ AONE C， HALVERSON L， HAMPTON T， et al. SRA: Description of the IE2 system used for MUC-7［C］// Proceedings of the 7th Message Understanding Conference. 1998.
［5］ MNIH A， HINTON G E. A scalable hierarchical distributed language model［C］// Proceedings of the 21st International Conference on Neural Information Processing Systems. 2008:1081-1088.
［6］ MIKOLOV T， SUTSKEVERI， CHEN K， et al. Distributed representations of words and phrases and their composi-tionality［C］// Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013:3111-3119.
［7］ PENNINGTON J， SOCHER R， MANNING C. Glove: Global vectors for word representation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing （EMNLP）. 2014:1532-1543.
［8］ BAHDANAU D， CHO K， BENGIO Y. Neural machine translation by jointly learning to align and translate［C］// Proceedings of 3rd International Conference on Learning Representations. 2015.
［9］ PETERS M E， NEUMANN M， IYYER M， et al. Deep contextualized word representations［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2018:2227-2237.
［10］ RADFORD A， NARASIMHAN K， SALIMANS T， et al. Improving 1anguage understanding by generative pretraining［J］. arXiv preprint arXiv:1802.05365， 2018.
［11］ DEVLIN J， CHANG M W， LEE K， et al. BERT: Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019:4171-4186.
［12］ YANG Z L， DAI Z H， YANG Y M， et al. XLNET: Generalized autoregressive pretraining for language understanding［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019:5753-5763.
［13］ DAI Z H， YANG Z L， YANG Y M， et al. Transformer-XL: Attentive language models beyond a fixed-length context［EB/OL］. arXiv preprint arXiv:1901.02860， 2019.
［14］ ZHONG Z X， CHEN D Q. A frustratingly easy approach for entity and relation extraction［C］// Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021:50-61.
［15］ XIE C H， LIANG J Q， LIU J P， et al. Revisiting the negative data of distantly supervised relation extraction［C］// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021:3572-3581.
［16］ SHANG Y M， HUANG H Y， MAo X L. OneRel: Joint entity and relation extraction with one module in one step［J］. Proceedings of the AAAI Conference on Artificial Intelligence. 2022，36（10）:11285-11293.
［17］ SHI X J， CHEN Z R， WANG H， et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. 2015:802-810.
［18］ JOZEFOWICZ R， VINYALS O， SCHUSTER M， et al. Exploring the limits of language modeling［J］. arXiv preprint arXiv:1602.02410， 2016.
［19］ LE CUN Y， BOSER B， DENKER J S， et al. Handwritten digit recognition with a back-propagation network［C］// Proceedings of the 2nd International Conference on Neural Information Processing Systems. 1989:396-404.
［20］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017:6000-6010.
［21］ HOWARD J， RUDER S. Universal language model fine-tuning for text classification［C］// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018:328-339.
［22］ LIU P J， SALEH M， POT E， et al. Generating Wikipedia by summarizing long sequences［J］. arXiv preprint arXiv:1801.10198， 2018.
［23］ JIANG X Z， LIANG Y B， CHEN W Z， et al. XLM-K: Improving cross-lingual language model pre-training with multilingual knowledge［J］. arXiv preprint arXiv:2109. 12573， 2021.
［24］ ZHANG Z Y， HAN X， LIU Z Y， et al. ERNIE: Enhanced language representation with informative entities［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019:1441-1451.
［25］ SUN Y， WANG S H， LI Y K， et al. ERNIE: Enhanced representation through knowledge integration［J］. arXiv preprint arXiv:1904.09223， 2019.
［26］ LIU X D， HE P C， CHEN W Z， et al. Multi-task deep neural networks for natural language understanding［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019:4487-4496.
［27］ SUN Y， WANG S H， LI Y K， et al. Ernie 2.0: A continual pre-training framework for language understanding［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2020，34（5）:8968-8975.
［28］ CUI Y M， CHE W X， LIU T， et al. Pre-training with whole word masking for chinese bert［J］. IEEE/ACM Transactions on Audio， Speech， and Language Processing， 2021，29:3504-3514.
［29］ LIU Y H， OTT M， GOYAL N， et al. Roberta: A robustly optimized bert pretraining approach［J］. arXiv preprint arXiv:1907.11692， 2019.
［30］ JOSHI M， CHEN D Q， LIU Y H， et al. Spanbert: Improving pre-training by representing and predicting spans［J］. Transactions of the Association for Computational Linguistics， 2020，8:64-77.
［31］ SONG K T， TAN X， QIN T， et al. MASS: Masked sequence to sequence pre-training for language generation［C］// Proceedings of the 36th International Conference on Machine Learning. 2019:5926-5936.
［32］ LAN Z Z， CHEN M D， GOODMAN S， et al. ALBERT: A lite BERT for self-supervised learning of language representations［J］. arXiv preprint arXiv:1909.11942， 2019.
［33］ JIAO X Q， YIN Y C， SHANG L F， et al. TinyBERT: Distilling BERT for natural language understanding［C］// Findings of the Association for Computational Linguistics: EMNLP 2020. 2020:4163-4174.
［34］ SANH V， DEBUT L， CHAUMOND J， et al. DistilBERT， a distilled version of BERT: Smaller， faster， cheaper and lighter［J］. arXiv preprint arXiv:1910.01108， 2019.
［35］ WEI J Q， REN X Z， LI X G， et al. NEZHA: Neural contextualized representation for Chinese language understanding［J］. arXiv preprint arXiv:1909.00204， 2019.
［36］ PETERS M E， NEUMANN M， LOGAN R， et al. Knowledge enhanced contextual word representations［C］// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019:43-54.
［37］ WANG J， LU W. Two are better than one: Joint entity and relation extraction with table-sequence encoders［C］ // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020:1706-1721.
［38］ WADDEN D， WENNBERG U， LUAN Y， et al. Entity， relation， and event extraction with contextualized span representations［C］// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019:5784-5789.
［39］ LUAN Y， WADDEN D， HE L H， et al. A general framework for information extraction using dynamic span graphs［C］// Proceedings of the 2019 Conference of the North. 2019:3036-3046.
［40］ COHEN A D， ROSENMAN S， GOLDBERG Y. Relation classification as two-way span-prediction［J］. arXiv preprint arXiv:2010.04829， 2020.
［41］ LI C， TIAN Y. Downstream model design of pre-trained language model for relation extraction task［J］. arXiv preprint arXiv:2004.03786， 2020.
［42］ TAO Q X， LUO X H， WANG H， et al. Enhancing relation extraction using syntactic indicators and sentential contexts［C］// 2019 IEEE 31st International Conference on Tools with Artificial Intelligence. 2019:1574-1580.
［43］ WEI Z P， SU J L， WANG Y， et al. A novel cascade binary tagging framework for relational triple extraction［C］// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020:1476-1488.
［44］ YE H B， ZHANG N Y， DENG S M， et al. Contrastive triple extraction with generative transformer［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2021，35（16）:14257-14265.
［45］ ZHOU W X， HUANG K， MA T Y， et al. Document-level relation extraction with adaptive thresholding and localized context pooling［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2021，35（16）: 14612-14620.
［46］ ZENG S， XU R X， CHANG B B， et al. Double graph based reasoning for document-level relation extraction［C］// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020:1630-1640.
［47］ DIXIT K， AL-ONAIZAN Y. Span-level model for relation extraction［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019:5308-5314.
［48］ SANH V， WOLF T， RUDER S. A hierarchical multi-task approach for learning embeddings from semantic tasks［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2019，33（1）:6949-6956.
［49］ ZHAO Y， WAN H Y， GAO J W， et al. Improving relation classification by entity pair graph［C］// Proceedings of the 11th Asian Conference on Machine Learning. 2019:1156-1171.
［50］ SOARES L B， FITZGERALD N， LING J， et al. Matching the blanks: Distributional similarity for relation learning［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019:2895-2905.
［51］ WU S C， HE Y F. Enriching pre-trained language model with entity information for relation classification［C］// Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019:2361-2364.
［52］ WANG H Y， TAN M， YU M， et al. Extracting multiple-relations in one-pass with pre-trained transformers［C］// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019:1371-1377.
［53］ ALT C， H[U]BNER M， HENNIG L. Improving relation extraction by pre-trained language representations［C］ // Automated Knowledge Base Construction （AKBC）. 2018.
［54］ LUO X K， LIU W J， MA M， et al. BiTT: Bidirectional tree tagging for joint extraction of overlapping entities and relations［J］. arXiv preprint arXiv:2008.13339， 2020.
［55］ SUN K， ZHANG R C， MENSAH S， et al. Recurrent interaction network for jointly extracting entities and classifying relations［C］// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020:3722-3732.
［56］ YAMADA I， ASAI A， SHINDO H， et al. LUKE: Deep contextualized entity representations with entity-aware self-attention［C］// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020:6442-6454.
［57］ YANG S M， YOO S Y， JEONG O R. DeNERT-KG: Named entity and relation extraction model using DQN， knowledge graph， and BERT［J］. Applied Sciences， 2020，10（18）:6429.
［58］ WANG R Z， TANG D Y， DUAN N， et al. K-Adapter: Infusing knowledge into pre-trained models with adapters［C］// Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021:1405-1418.
［59］ WANG X Z， GAO T Y， ZHU Z C， et al. KEPLER: A unified model for knowledge embedding and pre-trained language representation［J］. Transactions of the Association for Computational Linguistics， 2021，9（11）:176-194.
［60］ CHEN J， HOEHNDORF R， ELHOSEINY M， et al. Efficient long-distance relation extraction with DG-SpanBERT［J］. arXiv preprint arXiv:2004.03636， 2020.
［61］ XUE F Z， SUN A X， ZHANG H， et al. GDPNet: Refining Latent multi-view graph for relation extraction［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2021，35（16）:14194-14202.
［62］ PENG H， GAO T Y， HAN X， et al. Learning from context or names？ An empirical study on neural relation extraction［C］// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020:3661-3672.
［63］ SHI P， LIN J. Simple BERT models for relation extraction and semantic role labeling［J］. arXiv preprint arXiv:1904.05255， 2019.
［64］ HUANG K， WANG G T， MA T Y， et al. Entity and evidence guided relation extraction for DocRED［J］. arXiv preprint arXiv:2008.12283， 2020.
［65］ YE D M， LIN Y K， DU J J， et al. Coreferential reasoning learning for language representation［C］// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020:7170-7186.
［66］ NAN G， GUO Z， SEKULICI， et al. Reasoning with Latent structure refinement for document-level relation extraction［C］// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020:1546-1557.
［67］ JUNG W， SHIM K. Dual supervision framework for relation extraction with distant supervision and human annotation［C］// Proceedings of the 28th International Conference on Computational Linguistics. 2020:6411-6423.
［68］ TANG H Z， CAO Y N， ZHANG Z Y， et al. Hin: Hierarchical inference network for document-level relation extraction［C］// Proceedings of the 24th Pacific-Asia Conference， on Knowledge Discovery and Data Mining. 2020:197-209.
［69］ WANG H， FOCKE C， SYLVESTER R， et al. Fine-tune Bert for DocRED with two-step process［J］. arXiv preprint arXiv:1909.11898， 2019.
［70］ WANG Y C， YU B W， ZHANG Y Y， et al. TPLinker: Single-stage joint extraction of entities and relations through token pair linking［C］// Proceedings of the 28th International Conference on Computational Linguistics. 2020:1572-1582.

[1]	LIU Yunhai1, Feng Guang1, WU Xiaoting2, YANG Qun2 . Safety Helmet Wearing Detection Algorithm for Complex Construction Scenes [J]. Computer and Modernization, 2024, 0(12): 66-71.
[2]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[3]	CHEN Kai1, LI Yiting1, 2, QUAN Huafeng1. A River Discarded Bottles Detection Method Based on Improved YOLOv8 [J]. Computer and Modernization, 2024, 0(11): 113-120.
[4]	YANG Jun1, HU Wei1, ZHU Wenfu2. Visual SLAM Loop Closure Detection Algorithm Based on Improved MobileNetV3 [J]. Computer and Modernization, 2024, 0(10): 21-26.
[5]	WANG Yingying, HAO Xiao. Fine-grained Image Classification Based on Res2Net and Recursive Gated Convolution [J]. Computer and Modernization, 2024, 0(10): 74-79.
[6]	SHI Xingyu1, LI Qiang2, ZHUANG Li3, LIANG Yi3, WANG Qiulin3, CHEN Kai3, WU Chenzhou3, CHANG Sheng1. Object Detection Models Distillation Technique for Industrial Deployment [J]. Computer and Modernization, 2024, 0(10): 93-99.
[7]	ZHANG Ze1, ZHANG Jianquan2, 3, ZHOU Guopeng2, 3. Camera Module Defect Detection Based on Improved YOLOv8s [J]. Computer and Modernization, 2024, 0(09): 107-113.
[8]	YU Chenxi, GU Lin. Recognition and Warning of Elevator Abnormal Behavior Based on Human Skeleton [J]. Computer and Modernization, 2024, 0(09): 114-120.
[9]	CHENG Yazi1, LEI Liang1, 2, CHEN Han1, ZHAO Yiran1. Multi-scale Depth Fusion Monocular Depth Estimation Based on Transposed Attention [J]. Computer and Modernization, 2024, 0(09): 121-126.
[10]	CHENG Meng, LI Hao. Improved Deciduous Tree Nest Detection Method Based on YOLOv5s [J]. Computer and Modernization, 2024, 0(08): 24-29.
[11]	WANG Mengxi, LI Jun. Review of Fall Detection Technologies for Elderly [J]. Computer and Modernization, 2024, 0(08): 30-36.
[12]	YUE Youjun1, 2, ZHANG Yuankun1, ZHAO Hui1, 2, WANG Hongjun1, 2. Indoor Scene Recognition Method Based on Multi-scale Feature and Attention Module [J]. Computer and Modernization, 2024, 0(08): 37-42.
[13]	SHI Xianwei1, FAN Xin2. Semantic Segmentation of Video Frame Scene Based on Lightweight [J]. Computer and Modernization, 2024, 0(08): 49-53.
[14]	XU Xin’ai, LI Gang. An Image Generation Method of Classroom Expression Images [J]. Computer and Modernization, 2024, 0(08): 88-91.
[15]	ZHAO Xiaoming, PAN Ting, LIU Weifeng. Automated Drawing Psychoanalysis Based on Image Classification [J]. Computer and Modernization, 2024, 0(08): 92-97.