Medical Knowledge Extraction Based on BERT and Non-autoregressive

Abstract

Abstract: In order to avoid the problems of error accumulation and entity overlap caused by the pipeline entity relation extraction model， a joint extraction model based on BERT and Non-autoregressive is established for medical knowledge extraction. Firstly， with the help of the BERT pre-trained language model， the sentence code is obtained. Secondly， the Non-autoregressive method is proposed to achieve parallel decoding， extract the relationship type， extract entities according to the index of the subject and object entities， and obtain the medical triplet. Finally， we import the extracted triples into the Neo4j graph database and realize knowledge visualization. The dataset is derived from manual labeling of data in electronic medical records. The experimental results show that the F1 value， precision and recall based on BERT and non-autoregressive joint learning model are 0.92， 0.93 and 0.92， respectively. Compared with the existing model， the three evaluation indicators have been improved， indicating that the proposed method can effectively extract medical knowledge from electronic medical records.

Key words: joint learning, non-autoregressive, BERT, entity overlap, electronic medical record

YU Qing, MA Zhi-long, XU Chun. Medical Knowledge Extraction Based on BERT and Non-autoregressive[J]. Computer and Modernization, 2023, 0(01): 120-126.

References

［1］宁尚明，滕飞，李天瑞. 基于多通道自注意力机制的电子病历实体关系抽取［J］. 计算机学报， 2020，43（5）:916-929.
［2］贾辛洪，宋文爱，李伟岩，等. 慢阻肺知识图谱的构建研究与实现［J］. 小型微型计算机系统， 2020，41（7）:1371-1374.
［3］贾丽娜，陈恒，李冠宇. 基于注意力混合模型的中文医疗问答匹配［J］. 计算机应用与软件， 2021，38（11）:148-154.
［4］杜优. 基于信息融合的医疗影像辅助决策研究［J］. 智能计算机与应用， 2019，9（2）:91-95.
［5］ SOCHER R，HUVAL B，MANNING C D，et al. Semantic compositionality through recursive matrix-vector spaces ［C］// Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2012:1201-1211.
［6］ ZENG D J， LIU K，LAI S W， et al.Relation classification via convolutional deep neural network ［C］// Proceedings of the 25th International Conference on Computational Linguistics: Technical Paper. 2014: 2335-2344.
［7］冯云霞，衣鹏，韩正亮，等. 基于CNN-BGRU-CRF的中文电子病历实体抽取方法［J］. 计算机与现代化， 2020（11）:60-64.
［8］ CHIKKA V R， KARLAPALEM K. A hybrid deep learning approach for medical relation extraction［J］. arXiv preprint arXiv: 1806.11189， 2018.
［9］ UZUNER O， MAILOA J， RYAN R， et al. Semantic relations for problem-oriented medical records［J］. Artificial Intelligence in Medicine， 2010，50（2）: 63-73.
［10］ SUNIL K S，ASHISH A，KRISHNADEV O，et al. Relation extraction from clinical texts using domain invariant convolutional neural network［C］// Workshop on Biomedical Natural Language Processing. 2016:206-215.
［11］ DEVLIN J，CHANG M W，LEE K，et al. BERT：Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019，1: 4171-4186.
［12］ ZHANG X H，ZHANG Y Y，ZHANG Q，et al. Extracting comprehensive clinical information for breast cancer using deep learning methods［J］. International Journal of Medical Informatics， 2019，132:103985.1-103985.7.
［13］ CHRISTOPOULOU F， TRAN T， SAHU S K， et al. Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods［J］. Journal of the American Medical Informatics Association， 2020，27（1）:39-46.
［14］武小平，张强，赵芳，等. 基于BERT的心血管医疗指南实体关系抽取方法［J］. 计算机应用， 2021，41（1）:145-149.
［15］李丽双，袁光辉，刘晗喆. 基于位置降噪和丰富语义的电子病历实体关系抽取［J］. 中文信息学报， 2021，35（8）:89-97.
［16］张玉坤，刘茂福，胡慧君. 基于联合神经网络模型的中文医疗实体分类与关系抽取［J］. 计算机工程与科学， 2019，41（6）:1110-1118.
［17］ GETOOR L，TASKAR B. Global inference for entity and relation identification via a linear programming formulation［C］// Introduction to Statistical Relational Learning， 2007:553-580.
［18］ MIWA M， BANSAL M. End-to-end relation extraction using LSTMs on sequences and tree structures［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016，1:1105-1116.
［19］ ZHENG S C， WANG F BAO H Y， et al. Joint extraction of entities and relations based on a novel tagging scheme［C］// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017:1227-1236.
［20］ XU J，LI Z H，WEI Q，et al. Applying a deep learning-based sequence labeling approach to detect attributes of medical concepts in clinical text［J］. BMC Medical Informatics and Decision Making， 2019，19（S5）: Article Number 236.
［21］ PARSAEIMEHR E，FARTASH M，TORKESTANI J A. An enhanced deep neural network-based architecture for joint extraction of entity mentions and relations［J］. International Journal Of Fuzzy Logic And Intelligent Systems，2020，20（1）: 69-76.
［22］ WEI Z P，SU J L，WANG Y，et al. A novel cascade binary tagging framework for relational triple extraction［C］// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020:1476-1488.
［23］ WANG Y C，YU B W，ZHANG Y Y，et al. TPLinker: Single-stage joint extraction of entities and relations through token pair linking［C］// The 28th International Conference on Computational Linguistics. 2020:1572-1582.
［24］ ZENG X R，ZENG D J，HE S Z，et al. Extracting relational facts by an end-to-end neural model with copy mechanism［C］// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018:506-514.
［25］ CABOT P H，NAVIGLI R.REBEL：Relation extraction by end-to-end language generation［C］// Findings of the Association for Computational Linguistics：EMNLP 2021. 2021：2370-2381.
［26］贾浩，王煦，季佰军，等. 基于掩码机制的非自回归神经机器翻译［J］. 厦门大学学报（自然科学版）， 2021，60（4）:648-654.
［27］邓建国，张素兰，张继福，等. 监督学习中的损失函数及应用研究［J］. 大数据， 2020，6（1）:60-80.
［28］ NAYAK T，NG H T. Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction [C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020:8528-8535.

[1]	ZHENG Jiuchao, ZHAO Xinyuan. Entity Linking Method Based on Topics and Description Information [J]. Computer and Modernization, 2024, 0(12): 10-14.
[2]	MA Yu, YANG Yong, REN Ge, Palidan Tuerxun. Automated Essay Scoring Method Based on GCN and Fine Tuned BERT [J]. Computer and Modernization, 2024, 0(09): 33-37.
[3]	ZHAO Dun1, SHE Xuebing2, WU Changxing3. Named Entity Recognition in Field of Party Building Based on BERT-BiLSTM-CRF [J]. Computer and Modernization, 2024, 0(09): 91-94.
[4]	ZHENG Li-rui, XIAO Xiao-xia, ZOU Bei-ji, LIU Bin, ZHOU Zhan. Named Entity Recognition in Electronic Medical Record Based on BERT [J]. Computer and Modernization, 2024, 0(01): 87-91.
[5]	LIU Yu-peng, GE Yan, DU Jun-wei, CHEN Zhuo. Joint Extraction Method of Entities and Relations Based on FGM and Pointer Annotation [J]. Computer and Modernization, 2023, 0(11): 1-5.
[6]	TANG Shi-qi, ZHOU Rui-ping, XIE Shi-bin, LIU Meng-chi, XIAO Wen, . Cross-language Multi-label Sentiment Classification Based on Stacked Denoising AutoEncoder [J]. Computer and Modernization, 2023, 0(11): 6-12.
[7]	LI Shi-yue, MENG Jia-na, YU Yu-hai, LI Xue-ying, XU Ying-ao. Aspect Based Sentiment Analysis Model Based on Knowledge Enhancement [J]. Computer and Modernization, 2023, 0(10): 1-8.
[8]	WANG Hong-jie, XU Sheng-chao. Clustering Method of Cloud Platform Abnormal Transmission Data Based on Hilbert Similarity [J]. Computer and Modernization, 2023, 0(09): 27-31.
[9]	XIE Shi-chao, HUANG Wei, REN Xiang-hui. A Text Entity Linking Method Based on BERT [J]. Computer and Modernization, 2023, 0(02): 58-61.
[10]	ZHU Ya-jun, Yong Tso, Nyima Tashi, . Tibetan Medical Entity Recognition Based on Tibetan BERT [J]. Computer and Modernization, 2023, 0(01): 43-48.
[11]	HUANG Zhong-xiang, LI Ming. Text Classification Based on ALBERT Combined with Bidirectional Network [J]. Computer and Modernization, 2022, 0(10): 8-12.
[12]	ZHNAG Jun, QIU Long-long. A Text Classification Model Based on BERT and Pooling Operation [J]. Computer and Modernization, 2022, 0(06): 1-7.
[13]	CHEN Gang. Government Hotline Work-order Classification Fusing RoBERTa and Feature Extraction [J]. Computer and Modernization, 2022, 0(06): 21-26.
[14]	FAN Hai-wei, QIN Jia-jie, SUN Huan, ZHANG Li-miao, LU Xin-siyu. Traffic Accident Text Information Extraction Model Based on BERT and BiGRU-CRF Fusion [J]. Computer and Modernization, 2022, 0(05): 10-15.
[15]	GUO Tian-yu, YAN Rong-guo, FANG Xu-chen, XU Yu-ling, TAO Zheng-yi. Detection of R Wave Based on Hilbert Transform and Adaptive Threshold [J]. Computer and Modernization, 2022, 0(02): 114-119.