Research and Design on Topical Crawler Based on Analysis of Content and Link

Abstract

Abstract:

Abstract: In the aspect of grasping the topical webpage to the existing topical crawler algorithm, its accuracy is not high. This paper presents a topical webpage grasping method which based on evaluation of text content and webpage link. First it calculates the correlation of current webpage and theme, and then compares the correlation values with a given threshold to determine the current webpage is discarded or stored. At the same time the size of the correlation value also determines the priority of URL in the climbing link queue, this model takes into account the balance of topical webpage between accuracy and coverage. In the aspect of grasping topical webpage to design the new topical crawler algorithm, its accuracy has been improved to some extent.

Key words:

Key words: topical crawler, topical correlation, topical webpage

CLC Number:

TP393

SHU Ben, YIN Ke. Research and Design on Topical Crawler Based on Analysis of Content and Link[J]. Computer and Modernization, 2014, 0(4): 77-80.

References

［1］

刘运强. 垂直搜索引擎的研究与设计［J］. 计算机应用与软件， 2010,27(7):130132.

［2］付强. 基于Lucene的高效图书垂直搜索引擎的研究与实现［J］．太原师范学院学报， 2011，10(4)：103107．

［3］袁浩. 主题爬虫搜索Web页面策略的研究［D］. 长沙：中南大学， 2009.

［4］ Menczer F， Pant G， Ruiz M E， et al． Evaluating topic—driven Web crawlers［C］//Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval． New York，NY，USA， 2001:241249.

［5］ Bra D P， Post R. Searching for arbitrary information in the WWW: The fishsearch for mosaic［C］// The Second WWW Conference. Chicago： ACM Press， 1994:4551.

［6］ Herseovid M， Jaeov M， SMaarek Y. The sharksearch algorithm：An aplication:Tailored Web site mapping［J］. Computer Networks and ISDN System, 1998,30(6):317326.

［7］ Page L, Brin S, Motwani R. The PageRank Citation Ranking: Bring Order to the Web［R］. Technical Report SIDLWP19990120, Standford University, 1999.

［8］ Kleinberg J. Autboritative sources in a hyperlinked environment［J］. Journal of the ACM, 1999,46(5):604632.

［9］罗林波，陈绮，吴清秀. 基于SharkSearch和Hits算法的主题爬虫研究［J］. 计算机技术与发展， 2010,20(17):7679.

［10］宋宇，孟祥增. 基于改进FishSearch算法的多媒体检索［J］. 计算机工程， 2008,34(11):189190,193.

［11］谢治军. 垂直搜索引擎的主题网页抓取策略研究［D］. 重庆：重庆理工大学， 2012.

［12］金明珠，丁岳伟. 基于统计模型的主题爬虫的研究与实现［J］. 计算机工程与设计， 2010,31(16)：37003704.

［13］胡伟. 基于PageRank算法的主题爬虫研究与设计［D］. 武汉：武汉理工大学， 2012.

[1]	ZHENG Jiuchao, ZHAO Xinyuan. Entity Linking Method Based on Topics and Description Information [J]. Computer and Modernization, 2024, 0(12): 10-14.
[2]	ZHANG Simin, LIU Xinmei, YIN Junling, LI Baoling. PCB Defect Detection Method Based on Improved YOLOv7 [J]. Computer and Modernization, 2024, 0(12): 45-52.
[3]	LIU Baobao, YANG Jingjing, TAO Lu, WANG Heying . DSMSC Based on Attention Mechanism for Remote Sensing Image Scene Classification [J]. Computer and Modernization, 2024, 0(12): 72-77.
[4]	GU Yue, DENG Songfeng, SHEN Ji, MU Wentao, ZHAO Enqi. SAR Ship Detection Algorithm Based on Improved YOLOv8 [J]. Computer and Modernization, 2024, 0(12): 78-83.
[5]	WANG Yanyuan, MAO Zhengchong. Detection and Recognition Algorithms for Chinese and English Scene Text Images [J]. Computer and Modernization, 2024, 0(12): 84-90.
[6]	WAN Bing1, 2, 3, ZHAO Wentao4, PAN Duotao1, ZHAO Zhengtao2, 3, SUN Zhaoyang2, 3, YU Jiancheng2, 3 . Design of Semi-physical Simulation Test System for Unmanned Sailboat [J]. Computer and Modernization, 2024, 0(12): 91-99.
[7]	CHEN Liang, LI Cheng, YI Wei, XIONG Wei, WANG Xiaofan, TANG Haidong. Helmet Wearing Detection in Electric Power Field Based on#br# Millimeter-wave Radar and Visual Fusion [J]. Computer and Modernization, 2024, 0(12): 100-107.
[8]	XIAO Junbi, QIU Yi. SDN Failure Detection and Recovery Scheme Based on State-aware Data Plane [J]. Computer and Modernization, 2024, 0(12): 116-123.
[9]	ZHANG Tai1, YAN Zihao2, DUAN Jie2, ZHANG Zhihong2. Information Forwarding Strategy of Internet of Vehicles in Named Data Network [J]. Computer and Modernization, 2024, 0(11): 19-27.
[10]	DONG Chongchong, ZHAO Cong, WU You, ZHANG Lei, ZHANG Jiawen, LI Zhihao. Zero-trust Dynamic Evaluation Method for IoT Terminals [J]. Computer and Modernization, 2024, 0(11): 41-45.
[11]	ZHANG Kun1, ZHANG Yongwei1, WU Yongcheng1, ZHANG Xiaowen2, ZHAI Shichen2. An LLM-based Method for Automatic Construction of Equipment Failure Knowledge Graphs [J]. Computer and Modernization, 2024, 0(11): 46-53.
[12]	YE Xue, YANG Sheng, CHENG Kai, ZHU Feng. A Financial Knowledge Q&A Model for Power Enterprise Based on ChatGLM2-6B [J]. Computer and Modernization, 2024, 0(11): 54-63.
[13]	LI Junchao1, YOU Fei1, ZHANG Chao2, SU Lele2, GONG Yan2 . BiLSTM-Attention Prediction Model and Error Analysis #br# Based on Novel Multi-objective Coati Optimization Algorithm [J]. Computer and Modernization, 2024, 0(11): 70-76.
[14]	ZHANG Yu1, 2, LI Jing1, 2, MA Ming1, 2, WANG Zhongxiang1, 2, SUN Yan1, 2. YOLOLW: A Novel Lightweight Object Detection Model [J]. Computer and Modernization, 2024, 0(11): 91-98.
[15]	DONG Yuwen. Multi-scale Moving Object Detection Algorithm Based on Improved YOLOv7-tiny [J]. Computer and Modernization, 2024, 0(11): 99-105.