A Web News Extraction Method Based on Filtering Noise Wrapper

doi:10.3969/j.issn.1006-2475.2017.01.002

Abstract

Abstract: Extracting high-purity news from large Web pages, and stored in a structured form is the research foundation of the public opinion monitoring and topical updating. This paper proposes a Web news extraction method based on filtering noise wrapper. When inducing the wrapper, if the two strings don’t match each other, according to the threshold, we calculate the string tag path ratio of the strings to distinguish purity news from noise. At the same time, we propose two naive Bayes classifiers to extract the title and time of the news. Experimental results show that compared with other extraction technologies, the method in this paper has significant improvement in terms of accuracy and robustness. So it has greatly utility value.

Key words: information extraction, wrapper, Web news pages, algorithm of ACME;string tag path ratio, naive Bayes classifier

CLC Number:

TP391.1

SUN Meng, QU You-li. A Web News Extraction Method Based on Filtering Noise Wrapper[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2017.01.002.

References

[1] 罗超然,王纯,廖建新. 新闻网页内容抽取模块的设计与实现[J]. 电信技术, 2014(5):85-87.

[2] Chang C H, Kayed M, Girgis M R, et al. A survey of Web information extraction systems[J]. IEEE Transactions on Knowledge and Data Engineering, 2006,18(10):1411-1428.

[3] Grishman R. Information Extraction: Capabilities and Challenges[Z]. Notes Prepared for the 2012 International Winter School in Language and Speech Technologies, Rovira i Virgili University, Tarragona, Spain, 2012.

[4] Etzioni O, Banko M, Soderland S, et al. Open information extraction from the Web[J]. Communications of the ACM, 2008,51(12):68-74.

[5] Pol K, Patil N, Patankar S, et al. A survey on Web content mining and extraction of structured and semistructured data[C]// Proceedings of the 1st IEEE International Conference on Emerging Trends in Engineering and Technology. 2008:543-546.

[6] Crescenzi V, Mecca G, Merialdo P. RoadRunner: Towards automatic data extraction from large Web sites[C]// Proceedings of the 27th International Conference on Very Large Data Bases. 2001:109-118.

[7] Crescenzi V, Mecca G, Merialdo P. Automatic Web information extraction in the RoadRunner system[M]// Conceptual Modeling for New Information Systems Technologies. Springer Berlin Heidelberg, 2001:264-277.

[8] Crescenzi V, Mecca G, Merialdo P. Wrapping-oriented classification of Web pages[C]// Proceedings of the 2002 ACM Symposium on Applied Computing. 2002:1108-1112.

[9] Baroni M, Chantree F, Kilgarriff A, et al. Cleaneval: A competition for cleaning Web pages[C]// Proceedings of the 2008 International Conference on Language Resources and Evaluation. 2008:638-643.

[10] 李文奇,张忠能. 页面包装器自动生成的改进算法[J]. 计算机工程与应用, 2004,40(22):113-115.

[11] Wu Gongqing, Li Li, Hu Xuegang, et al. Web news extraction via path ratios[C]// Proceedings of the 22nd ACM International Conference on Information & Knowledge Management. 2013:2059-2068.

[12] 吴共庆. 基于标签路径特征的Web新闻内容抽取研究[D]. 合肥:合肥工业大学, 2012.

[13] 王中锋,王志海. 基于条件对数似然函数导数的贝叶斯网络分类器优化算法[J]. 计算机学报, 2012,35(2):364-374.

[14] 宫秀军. 贝叶斯学习理论及其应用研究[D]. 北京:中国科学院研究生院, 2002.

[1]	FAN Hai-wei, QIN Jia-jie, SUN Huan, ZHANG Li-miao, LU Xin-siyu. Traffic Accident Text Information Extraction Model Based on BERT and BiGRU-CRF Fusion [J]. Computer and Modernization, 2022, 0(05): 10-15.
[2]	ZHU Yan, ZHANG Li, WANG Yu. Named Entity Recognition on Chinese Electronic Medical Records Based on RoBERTa-WWM [J]. Computer and Modernization, 2021, 0(02): 51-55.
[3]	LI Pan1, LI Yi-guang2, XU Chun1. Hot Network Information Extraction Based on Key Nodes [J]. Computer and Modernization, 2019, 0(09): 60-.
[4]	LIANG Dong, YANG Yong-quan, WEI Zhi-qiang. Information Extraction of Web Pages Based on Support Vector Machine [J]. Computer and Modernization, 2018, 0(09): 21-.
[5]	ZHANG Jian-hua1, XIAO Zhong-zheng2. Opinion Target Extraction Method with Speech Rules and Dependency Parsing [J]. Computer and Modernization, 2016, 0(4): 16-20.
[6]	LIU Li-juan, ZHANG Yin, YANG Yi. Method of Web Information Extraction Based on Ontology Theory [J]. Computer and Modernization, 2015, 0(9): 90-94.
[7]	ZHANG Bing1， TANG Jin1,2， LUO Bin1,2. Web Page Title Real-time Extraction Method Based on Hyperlink and DOM Tree [J]. Computer and Modernization, 2015, 0(8): 84-88.
[8]	JIA Suimin, ZHANG Yu， ZHANG Tengfei. A Disaster Event Information Extraction Method Based on Preposition’s Usages [J]. Computer and Modernization, 2015, 0(7): 116-.
[9]	LIN Bo1, LIN Weijia2, GUO Jingyu1, DING Donghui2, HUANG Han2. Text Topic Extraction Based on Doublelinguisticfilter [J]. Computer and Modernization, 2015, 0(12): 7-.
[10]	HE Yuan. Research on Web Biological Information Extraction Method Based on Ontology [J]. Computer and Modernization, 2013, 1(5): 172-175.
[11]	LIU Jing-pei;LI Jiang;JI Wen-ping;PAN Peng-hui. Research on Text-oriented Event Information Extraction Method [J]. Computer and Modernization, 2012, 203(7): 198-201.
[12]	LI Hai-sheng;ZHOU Ping;HAN Meng-xiao;HUO Hong-yuan;CUI Yan-mei;GENG Ling-peng. Mineral Recognition Software Modules Development Based on Spectral Absorption Index [J]. Computer and Modernization, 2012, 198(2): 69-72.
[13]	QIAN Cheng;YANG Xiao-lan. Research on Transforming HTML into XML [J]. Computer and Modernization, 2011, 1(8): 39-41.
[14]	HOU Ming-liang. Research on Information Extraction Algorithm of Gaussian Noise Images Based on Fractional Order Differentiation [J]. Computer and Modernization, 2010, 1(9): 5-8.
[15]	ZHOU Xin;DENG Rong. Design and Research on Web Data Mining Model Based on XML [J]. Computer and Modernization, 2010, 1(11): 60-62.