Simulation of Web Crawler Detection Algorithm Based on Hidden Markov Model

doi:10.3969/j.issn.1006-2475.2017.04.025

Abstract

Abstract: In the construction and maintenance process of the website, in order to improve server efficiency, strengthen security and confidentiality, developers need to distinguish between human users and Web crawlers. However, some inappropriate or malicious designs make it difficult to detect crawlers. These crawlers not only increase the burden on the site, but also endanger the security of network. In order to solve the problem that it is difficult to detect crawlers, a detection algorithm based on behavior pattern is proposed, which uses hidden Markov model to describe the behavior patterns of different clients and uses Matlab simulation to achieve a highly accurate detection result. The simulation results show that the detection technology of hidden Markov model can detect Web crawler with high accuracy and low error rate.


Key words: Web crawler detection, behavior pattern, hidden Markov model, network security

CLC Number:

TP391.9

JU Xing-kong. Simulation of Web Crawler Detection Algorithm Based on Hidden Markov Model[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2017.04.025.

References

［1］孙立伟,何国辉,吴礼发. 网络爬虫技术的研究［J］. 电脑知识与技术, 2010,6(15):4112-4115.
［2］曾伟辉,李淼. 深层网络爬虫研究综述［J］. 计算机系统应用, 2008,17(5):122-126.
［3］ Doran D， Gokhale S S. Web robot detection techniques: Overview and limitations［J］. Data Mining & Knowledge Discovery, 2011,22(1):183-210
［4］ Stevanovic D, An A, Vlaji N C. Detecting Web crawlers from Web server access logs with data mining classifiers［C］// Proceedings of the 19th International Conference on Foundations of Intelligent Systems. 2011:483-489.
［5］李勇,韩亮. 主题搜索引擎中网络爬虫的搜索策略研究［J］. 计算机工程与科学, 2008,30(3):4-6.
［6］刘淑梅,夏亮,许南山. 主题搜索引擎网络爬虫搜索策略的研究与实现［J］. 计算机系统应用, 2010,19(3):49-52
［7］ Cho Younghee, Jeon Jinho, Lee Gyesung. Conceptual pattern matching of time series data using hidden Markov model［J］. Journal of the Korea Contents Association, 2008,8(5):44-51.
［8］ Almeida V， Daniel M E， Riedi R, et al. Analyzing Web robots and their impact on caching［C］// Proceedings of the 6th Web Caching and Content Delivery Workshop. 2001.
［9］ Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition［J］. Reading in Speech Recognition, 1990,77(2):267-296.
［10］范纯龙,袁滨,余周华,等. 基于陷阱技术的网络爬虫检测［J］. 计算机应用, 2010,30(7):1782-1784.
［11］周德懋,李舟军. 高性能网络爬虫研究综述［J］. 计算机科学, 2009,36(8):26-29.
［12］郭伟刚,鞠时光. 基于访问内容类型统计的Web Robot检测算法［J］. 计算机科学, 2005,32(8):179-181.
［13］郭伟刚,鞠时光. 一个基于事务分析的Web Robot检测算法［J］. 计算机应用, 2005,25(7):1580-1583.
［14］谭龙远. 基于领域的网络爬虫技术的研究与实现［D］. 武汉:武汉理工大学, 2009.
［15］Dikaiakos M D, Stassopoulou A, Papageorgiou L. An investigation of Web crawler behavior: Characterization and metrics［J］. Computer Communications, 2005,28(8):880-897.
［16］王艳红,周军. 基于Hadoop的网络爬虫技术研究［J］. 吉林工程技术师范学院学报, 2014,30(8):87-89.
［17］周立柱,林玲. 聚焦爬虫技术研究综述［J］. 计算机应用, 2005,25(9):1965-1969.
［18］Kwon S, Kim Y G, Cha S. Web robot detection based on pattern-matching technique［J］. Journal of Information Science, 2012,38(2):118-126.
［19］Zabihi M, Jahan M V, Hamidzadeh J. A density based clustering approach for Web robot detection［C］// 2014 4th International E-Conference on Computer & Knowledge Engineering. 2014:23-28.
［20］Tan Pang-Ning, Kumar V. Discovery of Web robot sessions based on their navigational patterns［J］. Data Mining and Knowledge Discovery, 2002,6(1):9-35.
［21］Chakrabarti S, Berg M V D， Dom B. Focused crawling: A new approach to topic-specific Web resource discovery［J］. Computer Networks, 1999,31(11-16):1623-1640.

[1]	WANG Hong-jie, XU Sheng-chao, YANG Bo, MAO Ming-yang, JIANG Jin-ling. Automatic Arrangement Method of Cloud Network Security Service Chain Based on SRv6 Technology [J]. Computer and Modernization, 2024, 0(01): 1-5.
[2]	YANG Jun, WANG Jin-lin, NI Hong, SHENG Yi-qiang, . Dynamic Transfer Method Based on Sensitivity in Industrial Control Network Anomaly Detection [J]. Computer and Modernization, 2023, 0(05): 46-51.
[3]	MAO Ming-yang, XU Sheng-chao. A New Active Defense Model for Complex Network Security [J]. Computer and Modernization, 2023, 0(04): 118-122.
[4]	PAN Yu-qing, ZHANG Su-ning, FENG Ren-jun, JING Dong-sheng. Intrusion Detection Method Based on Particle Swarm Optimization Combined with LightGBM [J]. Computer and Modernization, 2023, 0(04): 123-126.
[5]	CHU Su-hong, LIU Lei, . POF Protocol Parser [J]. Computer and Modernization, 2022, 0(09): 93-98.
[6]	WU Shui-ming, JI Zhi-yuan, WANG Zhen-yu, JING Dong-sheng. Power Information Network Intrusion Detection Algorithm Based on Dueling-DDQN [J]. Computer and Modernization, 2021, 0(12): 43-47.
[7]	ZHANG Wen-bo, HU Xi-ming, LI Peng, MA Miao, WANG Tao, . Design and Implementation of Security Experiment of Internet of Things Based on Smart Phone and Smart Home Appliances for New Engineering Disciplines#br# [J]. Computer and Modernization, 2021, 0(04): 109-116.
[8]	HAO Yi-ran, SHENG Yi-qiang, WANG Jing-lin, . Anomaly Detection of Network Traffic Based on t-SNE Dimensionality Reduction Preprocessing [J]. Computer and Modernization, 2021, 0(02): 109-116.
[9]	YAN Rui-an, ZHANG Li-chen. Intrusion Detection Based on Focal Loss and Convolutional Neural Network [J]. Computer and Modernization, 2021, 0(01): 65-69.
[10]	LIAO Zu-qi, LI Fei, ZHANG Peng-fei. CBR-based Emergency Response System and Method for Internet of Vehicle Security [J]. Computer and Modernization, 2020, 0(11): 109-116.
[11]	DAI Rui-feng. Model Construction of College Network Security System Based on SSL Virtual Technology [J]. Computer and Modernization, 2020, 0(08): 122-126.
[12]	CUI Wen-di, DUAN Peng-fei, ZHU Hong-qiang, LIU Na. Security Risk Assessmenton of Attack Graph and HMM Industrial Control Network [J]. Computer and Modernization, 2020, 0(07): 32-37.
[13]	LI He-ting, FENG Ren-jun, CHEN Hai-yan, JING Dong-sheng. Intrusion Detection Based on Heterogeneous Convolutional Neural Network [J]. Computer and Modernization, 2019, 0(10): 117-.
[14]	LIU Yi, LAN Shao-hua. Approach for Detecting Covert Timing Channels Based on One-class SVM [J]. Computer and Modernization, 2017, 0(6): 108-111+121.
[15]	WANG Hong-kai1, YU Xiao-wen2, ZHANG Xu-dong1. A Method of Security Risk Analysis for Network Based on Attack Detection and Vulnerability [J]. Computer and Modernization, 2017, 0(1): 96-100.