Recommendation Algorithm Model Based on DNN and Attention Mechanism

doi:10.3969/j.issn.1006-2475.2024.06.001

Abstract

Abstract: Abstract： In order to solve the defect of factorization machine in extracting high-order combination features and learn more useful feature information better， this paper attempts to use factorization machine to extract cross-feature and learn key feature information from low and high-order combination features by combining attention network， deep neural network， multi-head self-attention mechanism and other methods. Finally， the weighted fusion results were obtained according to the importance of the combination features of different orders， and the click-through rate of advertisements was estimated. The experiment was mainly carried out based on the advertising data set Criteo， and the analogy experiment was carried out with MovieLens data set to verify the effectiveness of the proposed algorithm model. The experimental results showed that compared with the benchmark model， in the two data sets， the AUC index increased by 2.32 percntage points and 0.4 percntage points.

Key words: Key words：factorization machine, neural network, attention network, extract cross-feature

CLC Number:

TP391

ZHOU Chao, CONG Xin, ZI Lingling, XIAO Guping. Recommendation Algorithm Model Based on DNN and Attention Mechanism[J]. Computer and Modernization, 2024, 0(06): 1-7.

References

［1］李欣琪，张学新. 人工智能时代的个性化推荐［J］.上海对外经贸大学学报， 2020，27（4）:90-99.
［2］张伟东. 推荐系统中实时在线推荐方法的研究［D］. 南京：南京邮电大学， 2022.
［3］冷亚军，陆青，梁昌勇. 协同过滤推荐技术综述［J］. 模式识别与人工智能， 2014，27（8）:720-734.
［4］张时俊. 基于矩阵分解的个性化推荐系统研究［D］. 长沙：湖南大学， 2016.
［5］ RENDLE S. Factorization machines［C］// 2010 IEEE International Conference on Data Mining. IEEE， 2010:995-1000.
［6］ CHENG H T， KOC L， HARMSEN J， et al. Wide & deep learning for recommender systems［C］// Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM， 2016:7-10.
［7］ QUEVEDO J R， MONTAÑÉS E， RANILLA J， et al. Ranked tag recommendation systems based on logistic regression［C］// 2010 International Conference on Hybrid Artificial Intelligence Systems. Springer， Berlin， Heidelberg， 2010:237-244.
［8］ PINKUS A. Approximation theory of the MLP model in neural networks［J］. ACTA Numerica， 1999，8:143-195.
［9］ GUO H F， TANG R M， YE Y M， et al. DeepFM: A facto-rization-machine based neural network for CTR prediction［J］. arXiv preprint arXiv:1703.04247， 2017.
［10］ WANG R X， FU B， FU G， et al. Deep & cross network for ad click predictions［C］// Proceedings of the ADKDD'17. ACM， 2017. DOI： 10.1145/3124749.3124754.
［11］ HE X， CHUA T S. Neural factorization machines for sparse predictive analytics［C］// Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM， 2017:355-364.
［12］ XIAO J， YE H， HE X N， et al. Attentional factorization machines: Learning the weight of feature interactions via attention networks［J］. arXiv preprint arXiv:1708.04617， 2017.
［13］皇甫晓瑛，钱惠敏，黄敏. 结合注意力机制的深度神经网络综述［J］. 计算机与现代化， 2023（2）:40-49
［14］ TOLSTIKHIN I O， HOULSBY N， KOLESNIKOV A， et al. MLP-mixer: An all-MLP architecture for vision［C］//Proceedings of the 2021 International Conference on Neural Information Processing Systems.ACM， 2021:24261-24272.
［15］于旭，何亚东，杜军威，等. 一种结合显式特征和隐式特征的开发者混合推荐算法［J］. 软件学报， 2022，33（5）:1635-1651.
［16］李建红，黄雅凡，王成军，等. 联合多层注意力网络矩阵分解的推荐算法［J］. 中文信息学报， 2022，36（3）:120-127.
［17］ ZHANG K， QIAN H， CUI Q， et al. Multi-interactive attention network for fine-grained feature learning in CTR prediction［C］// Proceedings of the 14th ACM International Conference on Web Search and Data Mining. ACM， 2021:984-992.
［18］ SONG W P， SHI C C， XIAO Z P， et al. AutoInt: Automatic feature interaction learning via self-attentive neural networks［C］// Proceedings of the 28th ACM International Conference on Information and Knowledge Management. ACM， 2019:1161-170.
［19］吴彬. 融合多头自注意力机制与新闻舆论特征的股票预测研究［D］. 南昌：江西财经大学， 2022.
［20］汪宏基. 基于深度学习的广告点击率预估研究［D］. 北京：北京交通大学， 2021.
［21］陈乔松，胡高浩，曹宏剑，等. 增强型注意力网络点击通过率预估方法［J］. 重庆邮电大学学报（自然科学版）， 2023，35（1）:147-155.
［22］张寅，朱桂熠，施天俊，等. 基于特征融合与注意力的遥感图像小目标检测［J］. 光学学报， 2022，42（24）:140-150.
［23］丁兵兵. 基于注意力机制的上下文主题对话生成［D］. 合肥：合肥工业大学， 2022.
［24］韩永亮. 基于特征交互学习的广告点击率预测模型研究［D］. 天津：天津理工大学， 2022.
［25］刘家铭. 基于深度学习和注意力机制的汽车广告推荐方法研究［D］. 大连：大连海事大学， 2022.
［26］ KINGMA D P， BA J. Adam: A method for stochastic optimization［J］. arXiv preprint arXiv:1412.6980， 2014.
［27］ SRIVASTAVA N， HINTON G， KRIZHEVSKY A， et al. Dropout: A simple way to prevent neural networks from overfitting［J］. The Journal of Machine Learning Research， 2014，15（1）:1929-1958.
［28］陈仁钦. 基于异构图注意力网络和循环神经网络的推荐算法［D］. 重庆：重庆大学， 2021.
［29］吴韦俊，李烨. 神经因子分解机推荐模型改进研究［J］. 软件导刊， 2020，19（4）:115-118.
［30］陈彬，张荣梅，张琦. DCFM：基于深度学习的混合推荐模型［J］. 计算机工程与应用， 2021，57（3）:150-155.
［31］王凯，沈艳. 基于历史行为与高低阶特征的点击率预估模型［J］. 软件导刊， 2023，22（5）:7-13.
［32］张子威，王鑫，朱文武. 图神经架构搜索综述［J］. 计算机学报，2023，46（7）：1532-1552.
［33］ DENG W， PAN J， ZHOU T， et al. Deeplight: Deep lightweight feature interactions for accelerating CTR predictions in ad serving［C］// Proceedings of the 14th ACM International Conference on Web Search and Data Mining. ACM，2021:922-930.

[1]	HE Sida, CHEN Pinghua. Intent-based Lightweight Self-Attention Network for Sequential Recommendation [J]. Computer and Modernization, 2024, 0(12): 1-9.
[2]	ZHANG Xiaodong1, BAI Guangzhi1, LI Min1, LI Haoyang2. Oil and Gas Well Production Prediction Model Based on Empirical Wavelet Transform [J]. Computer and Modernization, 2024, 0(12): 53-58.
[3]	LIU Baobao, YANG Jingjing, TAO Lu, WANG Heying . DSMSC Based on Attention Mechanism for Remote Sensing Image Scene Classification [J]. Computer and Modernization, 2024, 0(12): 72-77.
[4]	CHEN Liang, LI Cheng, YI Wei, XIONG Wei, WANG Xiaofan, TANG Haidong. Helmet Wearing Detection in Electric Power Field Based on#br# Millimeter-wave Radar and Visual Fusion [J]. Computer and Modernization, 2024, 0(12): 100-107.
[5]	MA Yu, YANG Yong, REN Ge, Palidan Tuerxun. Automated Essay Scoring Method Based on GCN and Fine Tuned BERT [J]. Computer and Modernization, 2024, 0(09): 33-37.
[6]	GAO Shuaipeng, WANG Yifan. Survey on Group-level Emotion Recognition in Images [J]. Computer and Modernization, 2024, 0(08): 98-107.
[7]	LAI Kun. Chinese Paper Invoice Text Recognition Method with Character Blurring [J]. Computer and Modernization, 2024, 0(08): 114-119.
[8]	ZHOU Xianxi, MU Li. News Long Text Classification Model Based on Improved TF-IDF and AGLCNN [J]. Computer and Modernization, 2024, 0(08): 120-126.
[9]	YANG Jiang1, SUN Xiaomei1, XU Tao2. Stock Price Prediction Based on Business Content to Construct Stock Association Relationships [J]. Computer and Modernization, 2024, 0(07): 21-25.
[10]	LIU Cunli1, LEI Zhanzhan2, ZHENG Ao2. Circular Convolutional Neural Network-based Defect Detection Method for#br# Drainage Pipe Networks [J]. Computer and Modernization, 2024, 0(07): 26-35.
[11]	WU Junkai, MAO Zhengchong. Laser Constant Current Source Controller Based on CDKF-RBFPID [J]. Computer and Modernization, 2024, 0(07): 100-105.
[12]	LI Shan, WANG Linna, GAO Dingjia, XUAN Haibo. Multi-layer Bank-enterprise Converged Network Based on Graph Neural Network [J]. Computer and Modernization, 2024, 0(05): 27-32.
[13]	ZHONG Hailong1, 2, HE Yueshun1, HE Linlin1, CHEN JIE1, TIAN Ming3, ZHENG Ruiyin4. Cost-sensitive Convolutional Neural Network for Encrypted Traffic Classification#br# #br# [J]. Computer and Modernization, 2024, 0(05): 55-60.
[14]	GAO Geng1, XIAO Fengli2, YANG Fei1. Recognition of Hypopigmented Skin Diseases Based on Improved MobileNetV3-Small [J]. Computer and Modernization, 2024, 0(05): 120-126.
[15]	YOU Jiajing1, 2, HE Yueshun1, HE Linlin1, ZHONG Hailong1, 2. Encryption Traffic Classification Method Based on AHP-CNN [J]. Computer and Modernization, 2024, 0(04): 83-87.