Image Animation Based on Generative Adversarial Networks

Abstract

Abstract: Anime-style images are highly simplified and abstract. In order to solve the problem of transforming real-world images into anime-style images, this paper proposes an image animation method based on generative adversarial networks. The generation network in this paper is like a U-Net fully convolutional structure. The input image is down-sampled first, and the shallow features are up-sampled by bilinear interpolation. The discriminant network uses Patch GAN and spectrum normalization. Semantic content loss and style loss are calculated separately to improve the stability of the network. Surface representation loss, structure representation loss, and texture representation loss are used to replace style loss to make the effect of generating animation pictures more controllable. We use train2014 for realistic images, and use the CelebA-HQ data set for face images. Experiments are performed on these data sets using this model. The experimental results show that the model in this paper can effectively complete the process of image animation and generate high-quality animation images.

Key words: deep learning, generative adversarial networks, image animation

ZHAI Hui-cong, ZHANG Ming, DENG Xing, WANG Li-qun. Image Animation Based on Generative Adversarial Networks[J]. Computer and Modernization, 2022, 0(07): 21-26.

References

［1］ ROSIN P, COLLOMOSSE J. Image and Video-Based Art-istic Stylisation［M］. Springer, 2012.
［2］ GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style［J］. arXiv preprint arXiv:1508.06576, 2015.
［3］ JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution［J］. arXiv preprint arXiv:1603.08155, 2016.
［4］ GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:2414-2423.
［5］ MIRZA M, OSINDERO S. Conditional generative adversarial nets［J］. arXiv preprint arXiv:1411.1784, 2014.
［6］ ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks［J］. arXiv preprint arXiv:1611.07004, 2018.
［7］ ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［J］. arXiv preprint arXiv:1703.10593, 2018.
［8］ CHEN Y, LAI Y K, LIU Y J. CartoonGAN: Generative adversarial networks for photo cartoonization［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018:9465-9474.
［9］ WANG X R, YU J Z. Learning to cartoonize using white-box cartoon representations［C］// Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition. 2020:1120-1130.
［10］LEE H Y, TSENG H Y, HUANG J B, et al. Diverse image-to-image translation via disentangled representations［C］// Proceedings of the 2018 European Conference on Computer Vision. 2018:35-51．
［11］BIAN Y A, LI X, LIU Y C, et al. Parallel coordinate descent Newton method for efficient L1-regularized loss minimization［J］. IEEE Transactions on Neural Networks and Learning Systems, 2019,30(11):3233-3245．
［12］ZHANG R, ISOLA P, EFROS A A. Colorful image colorization［C］// 2016 European Conference on Computer Vision. 2016:649-666.

［13］GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014,2:2672-2680.

［14］SIMONYAN K, ZISSERMAN A. Very deep convolutional network for large-scale image recognition［J］. arXiv preprint arXiv:1409.1556, 2014.
［15］MAO X D, LI Q, XIE H R, et al. Least squares generative adversarial networks［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. 2017:2813-2821.
［16］RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutional networks for biomedical image segmentation［C］// Proceedings of the 18th International Conference on Medical Image Computing and Computer Assisted Intervention. 2015:234-241.
［17］YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution［C］// Proceedings of the 2019 IEEE /CVF International Conference on Computer Vision. 2019:4470-4479.
［18］TONG T, LI G, LIU X J, et al. Image super-resolution using dense skip connections［C］// Proceedings of 2017 IEEE International Conference on Computer Vision. 2017:4809-4817.
［19］LAN Z Z, LIN M, LI X C, et al. Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015:204-212.
［20］MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral normalization for generative adversarial networks［J］. arXiv preprint arXiv:1803.05957, 2018.
［21］SHAHCHERAGHI Z, SEE J. On the effects of pre- and post-processing in video cartoonization with bilateral filters［C］// Proceedings of the 2013 IEEE International Conference on Signal and Image Processing Applications. 2013:37-42．
［22］HUANG X, BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. 2017:1510-1519.
［23］KINGMA D P, BA J L. Adam: A method for stochastic optimization［J］. arXiv preprint arXiv:1412.6980, 2014.
［24］KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation［C］// Proceedings of the 2018 IEEE International Conference on Learning Representations. 2018:324-330.
［25］WANG Z, BOVIK A C. A universal image quality index［J］. IEEE Signal Processing Letters, 2002,9(3):81-88.
［26］WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity［J］. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2004,13(4):600-612.
［27］HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium［C］// Proceedings of the 31th International Conference on Neural Information Processing Systems. 2017:6629-6640.

[1]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[2]	CHEN Kai1, LI Yiting1, 2, QUAN Huafeng1. A River Discarded Bottles Detection Method Based on Improved YOLOv8 [J]. Computer and Modernization, 2024, 0(11): 113-120.
[3]	YANG Jun1, HU Wei1, ZHU Wenfu2. Visual SLAM Loop Closure Detection Algorithm Based on Improved MobileNetV3 [J]. Computer and Modernization, 2024, 0(10): 21-26.
[4]	WANG Yingying, HAO Xiao. Fine-grained Image Classification Based on Res2Net and Recursive Gated Convolution [J]. Computer and Modernization, 2024, 0(10): 74-79.
[5]	SHI Xingyu1, LI Qiang2, ZHUANG Li3, LIANG Yi3, WANG Qiulin3, CHEN Kai3, WU Chenzhou3, CHANG Sheng1. Object Detection Models Distillation Technique for Industrial Deployment [J]. Computer and Modernization, 2024, 0(10): 93-99.
[6]	ZHANG Ze1, ZHANG Jianquan2, 3, ZHOU Guopeng2, 3. Camera Module Defect Detection Based on Improved YOLOv8s [J]. Computer and Modernization, 2024, 0(09): 107-113.
[7]	CHENG Yazi1, LEI Liang1, 2, CHEN Han1, ZHAO Yiran1. Multi-scale Depth Fusion Monocular Depth Estimation Based on Transposed Attention [J]. Computer and Modernization, 2024, 0(09): 121-126.
[8]	CHENG Meng, LI Hao. Improved Deciduous Tree Nest Detection Method Based on YOLOv5s [J]. Computer and Modernization, 2024, 0(08): 24-29.
[9]	WANG Mengxi, LI Jun. Review of Fall Detection Technologies for Elderly [J]. Computer and Modernization, 2024, 0(08): 30-36.
[10]	SHI Xianwei1, FAN Xin2. Semantic Segmentation of Video Frame Scene Based on Lightweight [J]. Computer and Modernization, 2024, 0(08): 49-53.
[11]	XU Xin’ai, LI Gang. An Image Generation Method of Classroom Expression Images [J]. Computer and Modernization, 2024, 0(08): 88-91.
[12]	GAO Shuaipeng, WANG Yifan. Survey on Group-level Emotion Recognition in Images [J]. Computer and Modernization, 2024, 0(08): 98-107.
[13]	HUANG Wendong, WANG Yifan. Survey on Multimodal Information Processing and Fusion Based on Modal Categories [J]. Computer and Modernization, 2024, 0(07): 47-62.
[14]	WU Li1, ZHANG Zhenghao2, GE Caicheng2, YU Jun2. Lane Line Detection Algorithm Based on Improved SCNN Network [J]. Computer and Modernization, 2024, 0(07): 87-92.
[15]	ZHANG Ke1, AI Zhongliang2, LIU Zhonglin3, GU Pingli1, LIU Xuelin4. Judicial Argumentation Understanding Method Based on Multiplet Loss [J]. Computer and Modernization, 2024, 0(06): 115-120.