基于注意力机制的人脸素描-照片图像生成网络

doi:10.3969/j.issn.1006-2475.2025.12.014

摘要/Abstract

摘要： 摘要：人脸素描-照片生成是图像生成的一个重要分支，在数字娱乐和公安刑侦等领域具有广泛应用。由于人脸素描图像只包含灰度信息，且丢失了人脸的大部分纹理细节，现有方法生成的人脸照片图像存在人脸结构缺失、细节信息不足和色彩失真等问题。针对上述问题，本文提出一种基于注意力机制的人脸素描-照片生成网络。首先，设计一种Vision Transformer 和U-Net相结合的生成器，有效提取人脸全局和局部特征，提升生成的人脸照片图像结构完整性。其次，构造一种改进的选择内核注意力模块，提升模型对细节信息的捕获能力，使生成的图像包含更多的人脸细节信息。最后，设计一种基于通道和像素注意力的判别器，增强生成对抗网络的对抗学习能力，减少生成人脸图像的颜色失真。通过与其他先进方法的主客观实验比较，本文提出的方法在人脸素描-照片图像生成任务上表现出更优的主观视觉效果和客观评价指标。在通用的CUHK、AR和XM2VTS人脸素描数据集上，本文方法的SSIM指标较次优值分别提升了11.6%、6.2%和4.5%，验证了本文方法的有效性。

关键词: 关键词：人脸素描-照片图像生成, 注意力机制； Transformer；生成对抗网络

Abstract: Abstract: Face sketch-photo synthesis is an important branch of image transformation, with broad applications in digital entertainment, public security, and criminal investigation. Since face sketch images only contain grayscale information and lose most of the texture details of the face, existing methods often suffer from structural deficiencies, insufficient detail information, and color distortion in the synthesized face photos. To address these issues, this paper proposes a face sketch-photo synthesis network based on an attention mechanism. First, this paper designs a generator by combining Vision Transformer and U-Net to effectively extract global and local facial features, improving the structural integrity of the generated face photos. Additionally, an improved selective kernel attention module is constructed to enhance the model’s ability to capture fine details, enabling the generated images to retain more facial texture information. Finally, this paper designs a discriminator based on channel and pixel-wise attention to strengthen the adversarial learning capability of the generative adversarial network (GAN), reducing color distortion in the synthesized face photo images. Through subjective and objective experiments comparing with other state-of-the-art methods, the proposed approach demonstrates superior performance in both visual quality and objective metrics for face sketch-photo synthesis. On the CUHK, AR, and XM2VTS face sketch datasets, the proposed method achieves 11.6%, 6.2%, and 4.5% improvements in SSIM metric over the second-best results, respectively, proving the effectiveness of the proposed method.

Key words: Key words: face sketch-photo synthesis, attention mechanism, Transformer, generative adversarial network

中图分类号:

中图分类号:TP391

姚丽1, 2, 占博思3, 万伟国3, 罗亦韬3, 杨宇仙4. 基于注意力机制的人脸素描-照片图像生成网络[J]. 计算机与现代化, 2025, 0(12): 97-106.

YAO Li1, 2, ZHAN Bosi3, WAN Weiguo3, LUO Yitao3, YANG Yuxian4. Face Sketch-photo Synthesis Network Based on Attention Mechanism[J]. Computer and Modernization, 2025, 0(12): 97-106.

参考文献

[1] 王清和,曹兵,朱鹏飞,等. 基于自判别循环生成对抗网络的人脸图像翻译[J]. 中国科学:信息科学, 2022,52(8):1447-1462.
[2] 员一南,曹林,康峻,等. 基于特征滤波的生成对抗网络素描人脸合成方法[J].计算机应用与软件, 2025,42(2):241-247.
[3] 翁丽芬,李晨阳,许华荣. 基于GAN的分步合成人脸素描生成算法[J]. 计算机辅助设计与图形学学报, 2023,35(9):1363-1373.
[4] GAO X B, ZHONG J J, LI J, et al. Face sketch synthesis algorithm based on E-HMM and selective ensemble[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2008,18(4):487-496.
[5] WANG X G, TANG X O. Face photo-sketch synthesis and recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008,31(11):1955-1967.
[6] CHANG L, ZHOU M, HAN Y J, et al. Face sketch synthesis via sparse representation[C]// Proceedings of the 2010 20th IEEE International Conference on Pattern Recognition. IEEE, 2010:2146-2149.
[7] GAO X B, WANG N N, TAO D C, et al. Face sketch-photo synthesis and retrieval using sparse representation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(8):1213-1226.
[8] LIU Q S, TANG X O, JIN H J, et al. A nonlinear approach for face sketch synthesis and recognition[C]// Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2005:1005-1010.
[9] 吉娜烨,柴秀娟,山世光,等. 局部回归模型驱动的人脸素描自动生成[J]. 计算机辅助设计与图形学学报, 2014,26(12):2232-2243.
[10] PENG C L, GAO X B, WANG N N, et al. Superpixel-based face sketch-photo synthesis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015,27(2):288-299.
[11] WAN W G, YANG Y, LEE H J. Generative adversarial learning for detail-preserving face sketch synthesis[J]. Neurocomputing, 2021,438:107-121.
[12] QUE Y, XIONG L, WAN W G, et al. Denoising diffusion probabilistic model for face sketch-to-photo synthesis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024,34(10):10424-10436.
[13] ZHANG L L, LIN L, WU X, et al. End-to-end photo-sketch generation via fully convolutional representation learning[C]// Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 2015:627-634.
[14] ZHANG D Y, LIN L, CHEN T S, et al. Content-adaptive sketch portrait generation by decompositional representation learning[J]. IEEE Transactions on Image Processing, 2016,26(1):328-339.
[15] MIRZA M, OSINDERO S. Conditional generative adversarial nets[J]. arXiv preprint arXiv:1411.1784, 2014.
[16] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv preprint arXiv: 1511.06434, 2015.
[17] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017:1125-1134.
[18] 葛延良,孙笑笑,王冬梅,等. 基于循环生成对抗网络的人脸素描合成网络设计[J]. 吉林大学学报(信息科学版), 2023, 41(1):76-83.
[19] CHEN C F, LIU W, TAN X, et al. Semi-supervised Cycle-GAN for face photo-sketch translation in the wild[J]. Computer Vision and Image Understanding, 2023,235. DOI: 10.1016/j.cviu.2023.103775.
[20] MELNIK A, MIASAYEDZENKAU M, MAKARAVETS D, et al. Face generation and editing with StyleGAN: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024,46(5):3557-3576.
[21] GUO M H, XU T X, LIU J J, et al. Attention mechanisms in computer vision: A survey[J]. Computational Visual Media, 2022,8(3):331-368.
[22] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018:7132-7141.
[23] LI X, WANG W H, HU X L, et al. Selective kernel networks[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2019:510-519.
[24] TSENG C H, LEE S J, FENG J N, et al. UPANets: Learning from the universal pixel attention networks[J]. Entropy, 2022,24(9). DOI: 10.3390/e24091243.
[25] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM, 2017:6000-6010.
[26] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE, 2017:2223-2232.
[27] FU H A, GONG M M, WANG C H, et al. Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2019:2427-2436.
[28] PARK T, EFROS A A, ZHANG R, et al. Contrastive learning for unpaired image-to-image translation[C]// Proceedings of the 16th European Conference on Computer Vision. Springer, 2020:319-345.
[29] ZHAO Y H, WU R H, DONG H. Unpaired image-to-image translation using adversarial consistency loss[C]// Proceedings of the 16th European Conference on Computer Vision. Springer, 2020:800-815.
[30] HAN J L, SHOEIBY M, PETERSSON L, et al. Dual contrastive learning for unsupervised image-to-image translation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2021:746-755.
[31] GOU Y, LI M, SONG Y, et al. Multi-feature contrastive learning for unpaired image-to-image translation[J]. Complex & Intelligent Systems, 2023,9(4):4111-4122.
[32] TORBUNOV D, HUANG Y, YU H W, et al. UVCGAN: Unet vision transformer cycle-consistent GAN for unpaired image-to-image translation[C]// Proceedings of the IEEE/ CVF Winter Conference on Applications of Computer Vision. IEEE, 2023:702-712.
[33] ZHAO C, CAI W L, YUAN Z. Spectral normalization and dual contrastive regularization for image-to-image translation[J]. The Visual Computer, 2025,41(1):129-140.
[34] GRASSI M, FAUNDEZ-ZANUY M. Face recognition experiments on the AR database[J]. NATO Security Through Science Series, E: Human and Societal Dynamics, 2007,18:275-284.
[35] MESSER K, MATAS J, KITTLER J, et al. XM2VTSDB: The extended M2VTS database[C]// Proceedings of the 2nd International Conference on Audio and Video-Based Biometric Person Authentication. Springer,1999,964:965-966.
[36] TANCHENKO A. Visual-PSNR measure of image quality[J]. Journal of Visual Communication and Image Representation, 2014,25(5):874-878.
[37] HORE A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]// Proceedings of the 2010 20th IEEE International Conference on Pattern Recognition. IEEE, 2010:2366-2369.
[38] ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018:586-595.
[39] ZHANG L, ZHANG L, MOU X Q, et al. FSIM: A feature similarity index for image quality assessment[J]. IEEE Transactions on Image Processing, 2011,20(8):2378-2386.
[40] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a nash equilibrium[J]. arXiv preprint arXiv:1706.08500, 2017.