Gaze Estimation Model Based on Hybrid Transformer

doi:10.3969/j.issn.1006-2475.2025.04.001

Abstract

Abstract: Combined CNN and Transformer， Transformer can gain the advantage of global feature information and improve the awareness of model context information， which can lead to improve model accuracy. A novel gaze estimation model RN-SA（ResNet-MHSA） based on a hybrid Transformer is proposed. In this model， part of the 3×3 spatial convolution layers in ResNet18 are replaced with a block composed of a 1×1 spatial convolution layer and MHSA（Multi-Head Self-Attention） layer， and the DropBlock mechanism is added to the model structure to increase the robustness of the model. Experimental results show that RN-SA model can improve the accuracy of the model while reducing the number of parameters compared with the current better model GazeTR-Hybrid， RN-SA model can improve the accuracy by 4.1% and 3.7% on EyeDiap and Gaze360 datasets， respectively， while the number of parameters is reduced by 15.8%. Therefore， the combination of CNN and Transformer can be effectively applied to gaze estimation tasks.

Key words: , gaze estimation, self-attention, MHSA, Transformer

CLC Number:

TP391.41

CHENG Zhang, LIU Dan, WANG Yanxia. Gaze Estimation Model Based on Hybrid Transformer [J]. Computer and Modernization, 2025, 0(04): 1-5.

References

［1］ RECASENS A， KHOSLA A， VONDRICK C， et al. Where are they looking？［C］// Proceedings of the 2015 International Conference on Neural Information Processing Systems. ACM， 2015：199-207.
［2］ KRAFKA K， KHOSLA A， KELLNHOFER P， et al. Eyetracking for everyone［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR）. IEEE， 2016：2176-2184.
［3］ FUNES-MORA K A， ODOBEZ J M. Gaze estimation in the 3D space using RGB-D sensors： Towards head-pose and user invariance［J］. International Journal of Computer Vision， 2016，118（2）：194-216.
［4］ DIAS P A， MALAFRONTE D， MEDEIROS H， et al. Gaze estimation for assisted living environments［C］// Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision. IEEE， 2020：279-288.
［5］ MODI N， SINGH J. Understanding online consumer behavior at e-commerce portals using eye-gaze tracking［J］. International Journal of Human-Computer Interaction， 2023，39（4）：721-742.
［6］ MASSÉ B， BA S， HORAUD R. Tracking gaze and visual focus of attention of people involved in social interaction［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018，40（11）：2711-2724.
［7］ CHETTAOUI N， ATIA A， BOUHLEL M S. Predicting students performance using eye-gaze features in an embodied learning environment［C］// 2022 IEEE Global Engineering Education Conference （EDUCON）. IEEE， 2022：704-711.
［8］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM， 2017：6000-6010.
［9］ DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16×16 words： Transformers for image recognition at scale［C］// 2021 International Conference on Learning Representations. ICLR， 2021. DOI：10.48550/arXiv.
2010.11929.
［10］ WANG W H， XIE E Z， LI X， et al. Pyramid vision Transformer： A versatile backbone for dense prediction without convolutions［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. IEEE，2021：548-558.
［11］ WU H P， XIAO B， CODELLA N， et al. CvT： Introducing convolutions to vision Transformers［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. IEEE， 2021：22-31.
［12］ CAO J L， PANG Y W， ANWER R M， et al. PSTR： End-to-end one-step person search with transformers［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2022：9448-9457.
［13］ ZHU X Z， SU W J， LU L W， et al. Deformable DETR： Deformable Transformers for end-to-end object detection［C］// 2021 International Conference on Learning Representations. ICLR， 2021. DOI： 10.48550/arXiv.2010.04159.
［14］ DAI Z G， CAI B L， LIN Y G， et al. UP-DETR： Unsupervised pre-training for object detection with Transformers［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021：1601
-1610.
［15］ WANG Y Q， XU Z L， WANG X L， et al. End-to-end video instance segmentation with Transformers［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021：8737-8746.
［16］ XUE F L， WANG Q C， GUO G D. Transfer： Learning relation-aware facial expression representations with Transformers［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. IEEE， 2021：3581-3590.
［17］ CHENG Y H， LU F. Gaze estimation using Transformer［C］// 2022 26th International Conference on Pattern Recognition （ICPR）. IEEE， 2022：3341-3347.
［18］ SRINIVAS A， LIN T Y， PARMAR N， et al. Bottleneck Transformers for visual recognition［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021：16514-16524.
［19］ ZHANG X C， SUGANO Y， FRITZ M， et al. Appearance-based gaze estimation in the wild［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern recognition. IEEE， 2015：4511-4520.
［20］ DENG H P， ZHU W J. Monocular free-head 30 gaze tracking with deep learning and geometry constraints［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE， 2017：3162-3171.
［21］ CHENG Y H， HUANG S Y， WANG F， et al. A coarse-to-fine adaptive network for appearance-based gaze estimation［C］// Proceedings of the 2020 AAAI Conference on Artificial Intelligence. AAAI， 2020，34（7）：10623-10630.
［22］ CAI X， ZENG J B， SHAN S G， et al. Source-free adaptive gaze estimation by uncertainty reduction［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2023：22035-22045.
［23］ LIU Z， LIN Y T， CAO Y， et al. Swin Transformer： Hierarchical vision Transformer using shifted windows［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. IEEE， 2021：9992-10002.
［24］ HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016：770-778.
［25］ GHIASI G， LIN T Y， LE Q V. Dropblock： A regularization method for convolutional networks［C］// Proceeding of the 32nd International Conference on Neural Information Processing Systems. ACM， 2018：10750-10760.
［26］ ZHANG X C， PARK S， BEELER T， et al. ETGth-XGaze： A large scale dataset for gaze estimation under extreme head pose and gaze variation［C］// Computer Vision ECCV 2020：16th European Conference. Springer， 2020：365-381.
［27］ FUNES-MORA K A， MONAY F， ODOBEZ J M. EYEDIAP： A database for the development and evaluation of gaze estimation algorithms from RGB and RGB-D cameras［C］// Proceedings of the 2014 Symposium on Eye Tracking Research and Applications. ACM， 2014：255-258.
［28］ KELLNHOFER P， RECASENS A， STENT S， et al. Gaze360： Physically unconstrained gaze estimation in the wild［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. IEEE， 2019：6911-6920.
［29］ ZHANG X C， SUGANO Y， FRITZ M， et al. It’s written all over your face： Full-face appearance-based gaze estimation［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE， 2017：2299-2308.
［30］ FISCHER T， CHANG H J， DEMIRIS Y. RT-GENE： Real-time eye gaze estimation in natural environments［C］// Proceedings of the 2018 European Conference on Computer Vision （ECCV）. Springer， 2018：339-357.
［31］ FARKHONDEH A， PALMERO C， SCARDAPANE S， et al. Towards self-supervised gaze estimation［J］. arXiv preprint arXiv：2203.10974， 2022.
［32］ ABDELRAHMAN A A， HEMPEL T， KHALIFA A， et al. L2CS-Net： Fine-grained gaze estimation in unconstrained environments［C］// 2023 8th International Conference on Frontiers of Signal Processing（ICFSP）. IEEE， 2023：98-102.

[1]	JI Zhengjie, WEI Linjing. ICS-ResNet： A Lightweight Network for Maize Leaf Disease Classification [J]. Computer and Modernization, 2025, 0(04): 19-28.
[2]	SUN Qining1, GUI Zhiming1, LIU Yanfang2, FAN Xinxin3, LU Yunfeng4. Graph Neural Network-based Multi-agent Reinforcement Learning for Adversarial Policy Detection Algorithm [J]. Computer and Modernization, 2025, 0(04): 42-49.
[3]	ZHANG Huiyu1, LIU Lei1, YAN Dongmei2, LIANG Chengqing3. UAV Path Planning Based on YOLO and PPO [J]. Computer and Modernization, 2025, 0(04): 50-55.
[4]	WU Yichuan. A3C Based Task Offloading and Resource Allocation Algorithm for Internet of Vehicles [J]. Computer and Modernization, 2025, 0(04): 56-62.
[5]	LIU Wenxin1, XU Wenhui1, CHEN Zhaoye1, GU Haiying1, WEN Cong2, YAO Yulong2, ZENG Xi2. Predictive Modeling of Ash Conveying in Thermal Power Plants Based on RFECV-XGBoost and SHAP [J]. Computer and Modernization, 2025, 0(04): 63-69.
[6]	ZHOU Jing1, 2, LIU Dunlong1, 2, SANG Xuejia1, 2, ZHANG Shaojie3, YANG Hongjuan3. Identification Method for Potential Debris Flow Basins in the Wenchuan Earthquake-Affected Area Based on CNN-KAN [J]. Computer and Modernization, 2025, 0(04): 70-76.
[7]	LI Kai, JIN Yunpeng, LI Haiyang, KONG Shasha, YANG Peng, FANG Chengwu, HUANG Xiangjie, HAN Yaosheng, LI Chunmei. AGP Calculation Methods in UAV Imagery Based on Image Segmentation [J]. Computer and Modernization, 2025, 0(04): 83-88.
[8]	WANG Lidan1, ZHAO Huaici2, PAN Duotao1, FANG Jian2, YUAN Decheng1. Infrared and Visible Image Fusion Based on Twin Axial-attention and Dual-discriminator Generative Adversarial Network [J]. Computer and Modernization, 2025, 0(04): 89-95.
[9]	MA Qi1, WEN Yudong1, LIANG Shangrong2, WANG Ke2. Cigarette Laser Code Recognition Method Based on DBNet and CRNN Fusion Model [J]. Computer and Modernization, 2025, 0(04): 96-102.
[10]	LI Zhihao, ZHAO Cong, WU You, CHEN Zechun, HE hang, DONG Chongchong. Construction and Migration Method of Zero Trust Architecture for Marketing System [J]. Computer and Modernization, 2025, 0(04): 119-126.
[11]	HOU Menghan, WEI Changfa. Construction of Depression Recognition Model Based on Multi-Feature Fusion [J]. Computer and Modernization, 2025, 0(03): 1-5.
[12]	TANG Rui1, WU Jianchao1, CHEN Jianbo1, CHAI Jiang1, WANG Qian1, HE Yuchen2. Improved YOLOv8s Algorithm Based on GiraffeDet for Transmission Line Icing Detection [J]. Computer and Modernization, 2025, 0(03): 6-11.
[13]	CAO Lu, DING Cangfeng, MA Lerong, YAN Zhaoyao, YOU Hao. Multilevel Joint Graph Embedding for Lipophilic Molecular Classification [J]. Computer and Modernization, 2025, 0(03): 12-21.
[14]	WANG Zeyu, HAN Jianning, HAO Guodong, YANG Run. Speech Enhancement Algorithm Based on Parallel Cascaded Time-frequency Conformer Generative Adversarial Network [J]. Computer and Modernization, 2025, 0(03): 22-28.
[15]	LEI Jiyue, SU Peng, NIE Yun, LIN Chuan. Review of Large Language Model Question Answering Systems for International Event Analysis [J]. Computer and Modernization, 2025, 0(03): 29-37.