Face Sketch-photo Synthesis Network Based on Attention Mechanism

doi:10.3969/j.issn.1006-2475.2025.12.014

Abstract

Abstract: Abstract: Face sketch-photo synthesis is an important branch of image transformation, with broad applications in digital entertainment, public security, and criminal investigation. Since face sketch images only contain grayscale information and lose most of the texture details of the face, existing methods often suffer from structural deficiencies, insufficient detail information, and color distortion in the synthesized face photos. To address these issues, this paper proposes a face sketch-photo synthesis network based on an attention mechanism. First, this paper designs a generator by combining Vision Transformer and U-Net to effectively extract global and local facial features, improving the structural integrity of the generated face photos. Additionally, an improved selective kernel attention module is constructed to enhance the model’s ability to capture fine details, enabling the generated images to retain more facial texture information. Finally, this paper designs a discriminator based on channel and pixel-wise attention to strengthen the adversarial learning capability of the generative adversarial network (GAN), reducing color distortion in the synthesized face photo images. Through subjective and objective experiments comparing with other state-of-the-art methods, the proposed approach demonstrates superior performance in both visual quality and objective metrics for face sketch-photo synthesis. On the CUHK, AR, and XM2VTS face sketch datasets, the proposed method achieves 11.6%, 6.2%, and 4.5% improvements in SSIM metric over the second-best results, respectively, proving the effectiveness of the proposed method.

Key words: Key words: face sketch-photo synthesis, attention mechanism, Transformer, generative adversarial network

CLC Number:

中图分类号:TP391

YAO Li1, 2, ZHAN Bosi3, WAN Weiguo3, LUO Yitao3, YANG Yuxian4. Face Sketch-photo Synthesis Network Based on Attention Mechanism[J]. Computer and Modernization, 2025, 0(12): 97-106.

References

[1] 王清和,曹兵,朱鹏飞,等. 基于自判别循环生成对抗网络的人脸图像翻译[J]. 中国科学:信息科学, 2022,52(8):1447-1462.
[2] 员一南,曹林,康峻,等. 基于特征滤波的生成对抗网络素描人脸合成方法[J].计算机应用与软件, 2025,42(2):241-247.
[3] 翁丽芬,李晨阳,许华荣. 基于GAN的分步合成人脸素描生成算法[J]. 计算机辅助设计与图形学学报, 2023,35(9):1363-1373.
[4] GAO X B, ZHONG J J, LI J, et al. Face sketch synthesis algorithm based on E-HMM and selective ensemble[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2008,18(4):487-496.
[5] WANG X G, TANG X O. Face photo-sketch synthesis and recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008,31(11):1955-1967.
[6] CHANG L, ZHOU M, HAN Y J, et al. Face sketch synthesis via sparse representation[C]// Proceedings of the 2010 20th IEEE International Conference on Pattern Recognition. IEEE, 2010:2146-2149.
[7] GAO X B, WANG N N, TAO D C, et al. Face sketch-photo synthesis and retrieval using sparse representation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(8):1213-1226.
[8] LIU Q S, TANG X O, JIN H J, et al. A nonlinear approach for face sketch synthesis and recognition[C]// Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2005:1005-1010.
[9] 吉娜烨,柴秀娟,山世光,等. 局部回归模型驱动的人脸素描自动生成[J]. 计算机辅助设计与图形学学报, 2014,26(12):2232-2243.
[10] PENG C L, GAO X B, WANG N N, et al. Superpixel-based face sketch-photo synthesis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015,27(2):288-299.
[11] WAN W G, YANG Y, LEE H J. Generative adversarial learning for detail-preserving face sketch synthesis[J]. Neurocomputing, 2021,438:107-121.
[12] QUE Y, XIONG L, WAN W G, et al. Denoising diffusion probabilistic model for face sketch-to-photo synthesis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024,34(10):10424-10436.
[13] ZHANG L L, LIN L, WU X, et al. End-to-end photo-sketch generation via fully convolutional representation learning[C]// Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 2015:627-634.
[14] ZHANG D Y, LIN L, CHEN T S, et al. Content-adaptive sketch portrait generation by decompositional representation learning[J]. IEEE Transactions on Image Processing, 2016,26(1):328-339.
[15] MIRZA M, OSINDERO S. Conditional generative adversarial nets[J]. arXiv preprint arXiv:1411.1784, 2014.
[16] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv preprint arXiv: 1511.06434, 2015.
[17] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017:1125-1134.
[18] 葛延良,孙笑笑,王冬梅,等. 基于循环生成对抗网络的人脸素描合成网络设计[J]. 吉林大学学报(信息科学版), 2023, 41(1):76-83.
[19] CHEN C F, LIU W, TAN X, et al. Semi-supervised Cycle-GAN for face photo-sketch translation in the wild[J]. Computer Vision and Image Understanding, 2023,235. DOI: 10.1016/j.cviu.2023.103775.
[20] MELNIK A, MIASAYEDZENKAU M, MAKARAVETS D, et al. Face generation and editing with StyleGAN: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024,46(5):3557-3576.
[21] GUO M H, XU T X, LIU J J, et al. Attention mechanisms in computer vision: A survey[J]. Computational Visual Media, 2022,8(3):331-368.
[22] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018:7132-7141.
[23] LI X, WANG W H, HU X L, et al. Selective kernel networks[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2019:510-519.
[24] TSENG C H, LEE S J, FENG J N, et al. UPANets: Learning from the universal pixel attention networks[J]. Entropy, 2022,24(9). DOI: 10.3390/e24091243.
[25] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM, 2017:6000-6010.
[26] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE, 2017:2223-2232.
[27] FU H A, GONG M M, WANG C H, et al. Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2019:2427-2436.
[28] PARK T, EFROS A A, ZHANG R, et al. Contrastive learning for unpaired image-to-image translation[C]// Proceedings of the 16th European Conference on Computer Vision. Springer, 2020:319-345.
[29] ZHAO Y H, WU R H, DONG H. Unpaired image-to-image translation using adversarial consistency loss[C]// Proceedings of the 16th European Conference on Computer Vision. Springer, 2020:800-815.
[30] HAN J L, SHOEIBY M, PETERSSON L, et al. Dual contrastive learning for unsupervised image-to-image translation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2021:746-755.
[31] GOU Y, LI M, SONG Y, et al. Multi-feature contrastive learning for unpaired image-to-image translation[J]. Complex & Intelligent Systems, 2023,9(4):4111-4122.
[32] TORBUNOV D, HUANG Y, YU H W, et al. UVCGAN: Unet vision transformer cycle-consistent GAN for unpaired image-to-image translation[C]// Proceedings of the IEEE/ CVF Winter Conference on Applications of Computer Vision. IEEE, 2023:702-712.
[33] ZHAO C, CAI W L, YUAN Z. Spectral normalization and dual contrastive regularization for image-to-image translation[J]. The Visual Computer, 2025,41(1):129-140.
[34] GRASSI M, FAUNDEZ-ZANUY M. Face recognition experiments on the AR database[J]. NATO Security Through Science Series, E: Human and Societal Dynamics, 2007,18:275-284.
[35] MESSER K, MATAS J, KITTLER J, et al. XM2VTSDB: The extended M2VTS database[C]// Proceedings of the 2nd International Conference on Audio and Video-Based Biometric Person Authentication. Springer,1999,964:965-966.
[36] TANCHENKO A. Visual-PSNR measure of image quality[J]. Journal of Visual Communication and Image Representation, 2014,25(5):874-878.
[37] HORE A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]// Proceedings of the 2010 20th IEEE International Conference on Pattern Recognition. IEEE, 2010:2366-2369.
[38] ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018:586-595.
[39] ZHANG L, ZHANG L, MOU X Q, et al. FSIM: A feature similarity index for image quality assessment[J]. IEEE Transactions on Image Processing, 2011,20(8):2378-2386.
[40] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a nash equilibrium[J]. arXiv preprint arXiv:1706.08500, 2017.