Named Entity Recognition on Chinese Electronic Medical Records Based on RoBERTa-WWM

Abstract

Abstract: Electronic Medical Records (EMRs) contain abundant information, such as clinical symptoms, diagnosis results and drug efficacy. Named Entity Recognition (NER) aims to extract named entities from unstructured texts. It is also the initial step to extract valuable information from the EMRs. This paper proposes a method to recognize named entities based on the RoBERTa-WWM (A Robustly Optimized BERT Pre-training Approach-Whole Word Masking). RoBERTa-WWM is a kind of pre-training model, which is utilized to generate semantic representations with prior knowledge. Compared with BERT (Bidirectional Encoder Representations from Transformers), the semantic representations generated by RoBERTa-WWM are more suitable for Chinese NER task because it masks the whole word during pre-training. The semantic representations are then inputted into Bidirectional Long Short-Term Memory (BiLSTM) and Conditional Random Field (CRF) models in turn. The experimental results show that this method can effectively improve the F1-score on “China Conference on Knowledge Graph and Semantic Computing 2019 (CCKS 2019)” dataset and improve the performance of NER in Chinese EMRs.

Key words: electronic medical records, named entity recognition, RoBERTa-WWM, information extraction

ZHU Yan, ZHANG Li, WANG Yu. Named Entity Recognition on Chinese Electronic Medical Records Based on RoBERTa-WWM[J]. Computer and Modernization, 2021, 0(02): 51-55.

References ［23］

［1］	杨锦锋,于秋滨,关毅,等. 电子病历命名实体识别和实体关系抽取研究综述［J］. 自动化学报, 2014,40(8):1537-1562.
［2］	梁书彤,郭茂祖,赵玲玲. 基于机器学习的医疗决策支持系统综述［J］. 计算机工程与应用, 2019,55(19):1-11.
［3］	SANG S T, YANG Z H, LIU X X, et al. GrEDeL: A knowledge graph embedding based method for drug discovery from biomedical literatures［J］. IEEE Access, 2018,7:8404-8415.
［4］	JAGANNATHA A N, YU H. Bidirectional RNN for medical event detection in electronic health records［C］// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016:473-482.
［5］	FRIEDMAN C, ALDERSON P O, AUSTIN J H M, et al. A general natural-language text processor for clinical radiology［J］. Journal of the American Medical Informatics Association, 1994,1(2):161-174.
［6］	GAIZAUSKAS R, DEMETRIOU G, HUMPHREYS K. Term recognition and classification in biological science journal articles［C］// Proceedings of the Computional Terminology for Medical and Biological Applications Workshop of the 2nd International Conference on NLP. 2000:37-44.
［7］	ZHOU G D, SU J. Named entity recognition using an HMM-based chunk tagger［C］// Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002:473-480.
［8］	JOZEFOWICZ R, ZAREMBA W, SUTSKEVER I. An empirical exploration of recurrent network architectures［C］// Proceedings of the 32nd International Conference on Machine Learning. 2015:2342-2350.
［9］	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017:6000-6010.
［10］	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019:4171-4186.
［11］	CUI Y M, CHE W Y, LIU T, et al. Revisiting pre-trained models for Chinese natural language processing［J］. arXiv: 2004.13922, 2020.
［12］	SONG M, YU H, HAN W S. Developing a hybrid dictionary-based bio-entity recognition technique［J］. BMC Medical Informatics and Decision Making, 2015,15(S1): Article No. S9.
［13］	LIANG J, XIAN X M, HE X J, et al. A novel approach towards medical entity recognition in Chinese clinical text［J］. Journal of Healthcare Engineering, 2017,2017: Article ID 4898963.
［14］	LI Z, ZHANG Q, LIU Y, et al. Recurrent neural networks with specialized word embedding for Chinese clinical named entity recognition［C］// CEUR Workshop Proceedings. 2017,1976:55-60.
［15］	XIA Y H, WANG Q. Clinical named entity recognition: ECUST in the CCKS-2017 shared task 2［C］// CEUR Workshop Proceedings. 2017,1976:43-48.
［16］	ZHU YY, WANG GY, KARLSSON B F. CAN-NER: Convolutional attention network for Chinese named entity recognition［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019:3384-3393.
［17］	PETERS M E, NEUMANN M, IYYER M, et al. Deep contextualized word representations［C］// Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2018:2227-2237.
［18］	HOCHREITER S, SCHMIDHUBER J. Long short-term memory［J］. Neural Computation, 1997,9(8):1735-1780.
［19］	LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data［C］// Proceedings of the 18th International Conference on Machine Learning. 2001:282-289.
［20］	马孟铖,杨晴雯,艾斯卡尔·艾木都拉,等. 基于词向量和条件随机场的中文命名实体分类［J］. 计算机工程与设计, 2020,41(9):2515-2522.
［21］	李文洁,张晴晴,张鹏远,等. 基于维特比算法的深度神经网络语音端点检测［J］. 重庆邮电大学学报(自然科学版), 2018,30(2):210-215.
［22］	柏兵,侯霞,石松. 基于CRF和BI-LSTM的命名实体识别方法［J］. 北京信息科技大学学报(自然科学版), 2018,33(6):27-33.
［23］	陈剑,何涛,闻英友,等. 基于BERT模型的司法文书实体识别方法［J］. 东北大学学报(自然科学版), 2020,41(10):1382-1387.

[1]	ZHENG Li-rui, XIAO Xiao-xia, ZOU Bei-ji, LIU Bin, ZHOU Zhan. Named Entity Recognition in Electronic Medical Record Based on BERT [J]. Computer and Modernization, 2024, 0(01): 87-91.
[2]	YU Tao, ZHANG Ying, Yong Tso, . Tibetan Named Entity Recognition Based on Small Sample Learning [J]. Computer and Modernization, 2023, 0(05): 13-19.
[3]	YU Peng, CHEN Yu-feng, XU Jin-an, ZHANG Yu-jie. Entity Recognition Method on EMR Based on Multi-task Learning [J]. Computer and Modernization, 2022, 0(09): 40-50.
[4]	FAN Hai-wei, QIN Jia-jie, SUN Huan, ZHANG Li-miao, LU Xin-siyu. Traffic Accident Text Information Extraction Model Based on BERT and BiGRU-CRF Fusion [J]. Computer and Modernization, 2022, 0(05): 10-15.
[5]	YANG Zheng, CAI Di, LI Hui-bin. Hierarchical Representation of Power Text Named Entity Recognition and Project-expert Matching [J]. Computer and Modernization, 2022, 0(05): 75-81.
[6]	LI Jian, ZHANG Ke-liang, TANG Liang, XIA Rong-jing, REN Jing-jing. Data Augmentation for Chinese Named Entity Recognition Task [J]. Computer and Modernization, 2022, 0(04): 1-6.
[7]	WANG Yun-qian, WANG Yi-song, CHEN Pan-feng, ZOU Long. Named Entity Recognition of Medicinal Plant Texts Integrated with Attention Mechanism [J]. Computer and Modernization, 2021, 0(11): 100-105.
[8]	ZHANG Cen-fang. Named Entity Recognition Algorithm Based on Active Learning [J]. Computer and Modernization, 2021, 0(07): 18-22.
[9]	XU Meng-di, WANG Jin-hua. Requirements Document Named Entity Recognition Based on Deep Learning and Grammatical Regulations [J]. Computer and Modernization, 2021, 0(01): 105-110.
[10]	LI Pan1, LI Yi-guang2, XU Chun1. Hot Network Information Extraction Based on Key Nodes [J]. Computer and Modernization, 2019, 0(09): 60-.
[11]	LIANG Dong, YANG Yong-quan, WEI Zhi-qiang. Information Extraction of Web Pages Based on Support Vector Machine [J]. Computer and Modernization, 2018, 0(09): 21-.
[12]	ZHANG Lei. Research on Named Entity Recognition Method in Specific Fields [J]. Computer and Modernization, 2018, 0(03): 60-.
[13]	ZHANG Jian, WU Qing, YANG Xin-yi, WANG Bin-cong, WU Xuan-wei, XU Xiang-ying, LYU Qiang. Chinese Agricultural Named Entity Recognition Based on Conditional Random Fields [J]. Computer and Modernization, 2018, 0(01): 123-126.
[14]	SUN Meng, QU You-li. A Web News Extraction Method Based on Filtering Noise Wrapper [J]. Computer and Modernization, 2017, 0(1): 5-12.
[15]	ZHANG Jian-hua1, XIAO Zhong-zheng2. Opinion Target Extraction Method with Speech Rules and Dependency Parsing [J]. Computer and Modernization, 2016, 0(4): 16-20.