Facial Sketch Image Conversion Based on CycleGAN and Attention Mechanism

doi:10.3969/j.issn.1006-2475.2025.09.009

Abstract

Abstract:
Abstract： In recent years， because of its demand in law enforcement， criminal and entertainment fields， face sketch-photo synthesis has become a research hotspot. As deep learning model without paired image supervision， CycleGAN is good at cross-domain image conversion， providing a powerful tool for efficient conversion between sketches and photos. In view of the difficulty of collecting a large number of pairs of face images and sketch images， and the problems of fuzzy and low definition image details in face sketch image generation， an improved CycleGAN model is proposed. In this paper， the self-attention mechanism is introduced into the residual block of the ResNet architecture generator in the CycleGAN model， so that the CycleGAN generator model can learn the features of different channels and the importance of different regions in the face image more effectively， and automatically focus on the important regions of facial features， such as eyes， nose， mouth， etc.， during image processing. At the same time， the edge clarity and integrity of the sketch are increased， so as to improve the quality of the generated face sketch image. The proposed model is implemented on the datasets CUHK and FS2K. The structural similarity， peak signal-to-noise ratio and multi-scale structural similarity are 0.7741， 11.7451 and 0.8504 respectively on CUHK and 0.7049， 13.2745 and 0.7970 respectively on FS2K. These results outperformed the comparison models of CycleGAN， Pix2Pix， MUNIT， and DCLGAN. According to the comparison experiment and subjective vision， the proposed model can effectively complete the process of face sketching and generate higher quality face sketching images.

Key words: Key words： CycleGAN； generative adversarial network； attention mechanism； residual network ,

CLC Number:

中图分类号：TP391

LIN Ruizi, YAO Da, DAI Xin, SHEN Guoyu, WANG Jiahui, WAN Weiguo. Facial Sketch Image Conversion Based on CycleGAN and Attention Mechanism[J]. Computer and Modernization, 2025, 0(09): 61-66.

References

［1］ YANG M， CHEN W， WANG Z， et al. Face sketching in the wild： A survey［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2021，43（5）：1600-1619.
［2］ MIRZA M， OSINDERO A S， BETHGE M， Image style transfer using convolutional neural networks［C］// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016：2414-2423.
［3］ CHOI Y， UH Y， YOO J， HA J W. StarGAN v2： Diverse image synthesis for multiple domains［C］// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. IEEE， 2020：8185-8194.
［4］ ISOLA P， ZHU J Y， ZHOU T H， et al. Image-to-image translation with conditional adversarial networks［J］. arXiv preprint arXiv：1611.07004， 2018.
［5］ ZHU J Y， PARK T， ISOLA P， et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［J］. arXiv preprint arXiv：1703.10593， 2018.
［6］ GATYS L A， ECKER A S， BETHGE M. Image style transfer using convolutional neural networks［C］// Proceedings of 2016 1EEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016：2414-2423.
［7］ GOODFELLOW L， POUGET-ABADIE J， MIRZA M， et al. Generative adversarial nets［C］// Advances on Neurainformation Processing Systems. Springer， 2014：2672-268.
［8］ TAN M， LE Q V. EfficientNet： Rethinking model scaling for convolutional neural networks［C］// Proceedings of the 36th International Conference on Machine Learning. PMLR， 2019：6105-6114.
［9］ IOFFE S. Batch renormalization： Towards reducing minibatch dependence in batch-normalized models［C］// Advances on Neural Information Processing Systems.Springer， 2017：1945-1953.
［10］ RAMACHANDRAN P， ZOPH B， LE Q V. Searching for activation functions［J］. arXiv preprint arXiv：1710.05941， 2017.
［11］ HE K M， ZHANG X Y， REN S Q， et al. Improved residual networks（iResNet）［J］. arXiv preprint arXiv：2002.10543， 2020.
［12］ WANG T C， LIU M Y， ZHU J Y， et al. High-resolution image synthesis and semantic manipulation with conditional GANs［C］// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2018：8798-8807.
［13］ ISOLA P， ZHU J Y， ZHOU T， et al. Image-to-image translation with conditional adversarial networks［C］// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2017：1125-1134.
［14］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［J］. arXiv preprint arXiv：1706.03762， 2017.
［15］ HU J， SHEN L， ALBANIE S， et al. Squeeze-and-excitation networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2019，42（8）：2011-2023.
［16］ FU J， LIU J， TIAN H J， et al. Dual attention network for scene segmentation［C］// Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2019：3146-3154.
［17］ WANG X， GIRSHICK R， GUPTA A， et al. Non-local neural networks［C］// Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2018：7794-7803.
［18］ ZHANG H， GOODFELLOW I， METAXAS D， et al. Self-attention generative adversarial networks［C］// International Conference on Machine Learning. PMLR， 2019：7354-7363.
［19］ WANG M Q， LU S Y， ZHU D Y， et al. A high-speed and low-complexity architecture for Softmax function in deep learning［C］// 2018 IEEE Asia Pacific Conference on Circuits and Systems （APCCAS）. IEEE， 2018：223-226.
［20］ WANG X G， TANG X O. Face photo-sketch synthesis and recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence （PAMI）， 2009，31（11）：1955-1967.
［21］ PING D F， ZILING H， PENG Z， et al. Facial-sketch synthesis： A new challenge［J］. Machine Intelligence Research， 2022，19（4）：257-287.
［22］ HAN J L， SHOEIBY M， PETERSSON L， et al. Dual contrastive learning for unsupervised image-to-image translation［J］. arXiv preprint arXiv：2104.07689， 2021.
［23］ HUANG X， LIU M Y， BELONGIE S， et al. Multimodal unsupervised image-to-image translation［J］. arXiv preprint arXiv：1804.04732， 2018.
［24］ KINGMA D P， BA J L. Adam： A method for stochastic optimization［J］. arXiv preprint arXiv：1412.6980， 2014.
［25］ WANG Z， BOVIK A C. A universal image quality index［J］. IEEE Signal Processing Letters， 2002，9（3）：81-88.
［26］ WANG Z， BOVIK A C， SHEIKH H R， et al. Image quality assessment： From error visibility to structural similarity［J］. IEEE Transactions on Image Processing， 2004，13（4）：600-612.
［27］ Richter T， Kim K J. A MS-SSIM optimal JPEG 2000 encoder［C］// Proceedings of the 2009 Data Compression Conference. ACM， 2009：401-410.

[1]	LI Xiongqing1, 2, PENG Mingtian1, 2, LI Yong1, 2, WANG Junfei1, 2, LIU Dezhi1, 3, BIAN Yuxuan1, 3, CHAI Yuelin1, 3, LIU Yuntao1, 3. Survey on Bundle Recommendation Algorithms [J]. Computer and Modernization, 2025, 0(09): 1-13.
[2]	GUO Jinhao, WANG Fengping, WANG Haoqi. PB-YOLOv7 Pedestrian Detection Method for Dense Scenes [J]. Computer and Modernization, 2025, 0(09): 14-19.
[3]	YAO Jie1, YIN Honghai1, WANG Dahai1, LI Runzi2, ZHANG Qianwen2, GUO Yu2. Target Pose Estimation Methods for Distribution Network Bypass Operation Robots [J]. Computer and Modernization, 2025, 0(09): 20-26.
[4]	ZHANG Jingying1, GENG Lin2, LIU Ningzhong2. Low-data Fine-grained Image Classification Based on Self-distillation and Self-attention Enhancement [J]. Computer and Modernization, 2025, 0(09): 27-34.
[5]	LUO Deyan, XU Yang, ZUO Fengyun, ZHANG Yongdan. Lightweight Garbage Classification and Detection of Improved YOLOv8 [J]. Computer and Modernization, 2025, 0(09): 35-42.
[6]	JIA Tao1, WU Yuechao2, LYU Yang2, FU Wenlong3. Defect Detection of Photovoltaic Panel Based on Improved YOLOv8-EDD [J]. Computer and Modernization, 2025, 0(09): 43-49.
[7]	WANG Song1, LI Yuanzhi2, CHEN Wei1, BIAN Yuxiang3. Quantum Identity Authentication Protocol Based on Quantum Secure Direct Communication [J]. Computer and Modernization, 2025, 0(09): 50-54.
[8]	JI Xiulan. User Device Security Authentication Method for Cloud Computing Data Transmission [J]. Computer and Modernization, 2025, 0(09): 55-60.
[9]	JI Xingyu1, HUANG Chenrong2, YAO Juncai2, WANG Kai1, GU Mingjie1. Insulator Defect Detection Based on Complex Environment [J]. Computer and Modernization, 2025, 0(09): 67-72.
[10]	ZHU Lulu, GU Lin. Arbitrary Style Transfer Method with Multi-scale Semantic Adaptation [J]. Computer and Modernization, 2025, 0(09): 73-78.
[11]	YANG Tianshun1, SONG Huansheng1, LIANG Haoxiang2, LIU Haonan1, MA Xinzhou1, SUN Shijie1, ZHANG Shaoyang1. Multimodal Tunnel Fire Detection Based on Temporal Features of Video [J]. Computer and Modernization, 2025, 0(09): 79-89.
[12]	LI Su, LI Dequan, QING Yu. Single View 3D Hair Modeling Based on Mask Constraints [J]. Computer and Modernization, 2025, 0(09): 90-96.
[13]	LIU Cheng, FENG Guang. CNN-BiLSTM and LightGBM Stock Prediction Based on Dual Attention Mechanism [J]. Computer and Modernization, 2025, 0(09): 97-103.
[14]	SHANG Boxiang1, GUO Xiaoyan2, ZHENG Jian1, SUN Xianfan2. Joint Knowledge Extraction for Cloud-edge Collaborative Multi-source Transmission Data [J]. Computer and Modernization, 2025, 0(09): 104-108.
[15]	CHENG Mengyuan, LI Yanhui, LYU Tianci, ZHAO Yuxin, HUANG Chen. FedLDP： Efficient Federated Learning with Localized Differential Privacy [J]. Computer and Modernization, 2025, 0(09): 109-118.