Image Animation Based on Generative Adversarial Networks

Abstract

Abstract: Anime-style images are highly simplified and abstract. In order to solve the problem of transforming real-world images into anime-style images, this paper proposes an image animation method based on generative adversarial networks. The generation network in this paper is like a U-Net fully convolutional structure. The input image is down-sampled first, and the shallow features are up-sampled by bilinear interpolation. The discriminant network uses Patch GAN and spectrum normalization. Semantic content loss and style loss are calculated separately to improve the stability of the network. Surface representation loss, structure representation loss, and texture representation loss are used to replace style loss to make the effect of generating animation pictures more controllable. We use train2014 for realistic images, and use the CelebA-HQ data set for face images. Experiments are performed on these data sets using this model. The experimental results show that the model in this paper can effectively complete the process of image animation and generate high-quality animation images.

Key words: deep learning, generative adversarial networks, image animation

ZHAI Hui-cong, ZHANG Ming, DENG Xing, WANG Li-qun. Image Animation Based on Generative Adversarial Networks[J]. Computer and Modernization, 2022, 0(07): 21-26.

References ［27］

［1］	ROSIN P, COLLOMOSSE J. Image and Video-Based Art-istic Stylisation［M］. Springer, 2012.
［2］	GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style［J］. arXiv preprint arXiv:1508.06576, 2015.
［3］	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution［J］. arXiv preprint arXiv:1603.08155, 2016.
［4］	GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:2414-2423.
［5］	MIRZA M, OSINDERO S. Conditional generative adversarial nets［J］. arXiv preprint arXiv:1411.1784, 2014.
［6］	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks［J］. arXiv preprint arXiv:1611.07004, 2018.
［7］	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks［J］. arXiv preprint arXiv:1703.10593, 2018.
［8］	CHEN Y, LAI Y K, LIU Y J. CartoonGAN: Generative adversarial networks for photo cartoonization［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018:9465-9474.
［9］	WANG X R, YU J Z. Learning to cartoonize using white-box cartoon representations［C］// Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition. 2020:1120-1130.
［10］	LEE H Y, TSENG H Y, HUANG J B, et al. Diverse image-to-image translation via disentangled representations［C］// Proceedings of the 2018 European Conference on Computer Vision. 2018:35-51．
［11］	BIAN Y A, LI X, LIU Y C, et al. Parallel coordinate descent Newton method for efficient L1-regularized loss minimization［J］. IEEE Transactions on Neural Networks and Learning Systems, 2019,30(11):3233-3245．
［12］	ZHANG R, ISOLA P, EFROS A A. Colorful image colorization［C］// 2016 European Conference on Computer Vision. 2016:649-666.
［13］	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014,2:2672-2680.
［14］	SIMONYAN K, ZISSERMAN A. Very deep convolutional network for large-scale image recognition［J］. arXiv preprint arXiv:1409.1556, 2014.
［15］	MAO X D, LI Q, XIE H R, et al. Least squares generative adversarial networks［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. 2017:2813-2821.
［16］	RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutional networks for biomedical image segmentation［C］// Proceedings of the 18th International Conference on Medical Image Computing and Computer Assisted Intervention. 2015:234-241.
［17］	YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution［C］// Proceedings of the 2019 IEEE /CVF International Conference on Computer Vision. 2019:4470-4479.
［18］	TONG T, LI G, LIU X J, et al. Image super-resolution using dense skip connections［C］// Proceedings of 2017 IEEE International Conference on Computer Vision. 2017:4809-4817.
［19］	LAN Z Z, LIN M, LI X C, et al. Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015:204-212.
［20］	MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral normalization for generative adversarial networks［J］. arXiv preprint arXiv:1803.05957, 2018.
［21］	SHAHCHERAGHI Z, SEE J. On the effects of pre- and post-processing in video cartoonization with bilateral filters［C］// Proceedings of the 2013 IEEE International Conference on Signal and Image Processing Applications. 2013:37-42．
［22］	HUANG X, BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. 2017:1510-1519.
［23］	KINGMA D P, BA J L. Adam: A method for stochastic optimization［J］. arXiv preprint arXiv:1412.6980, 2014.
［24］	KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation［C］// Proceedings of the 2018 IEEE International Conference on Learning Representations. 2018:324-330.
［25］	WANG Z, BOVIK A C. A universal image quality index［J］. IEEE Signal Processing Letters, 2002,9(3):81-88.
［26］	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity［J］. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2004,13(4):600-612.
［27］	HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium［C］// Proceedings of the 31th International Conference on Neural Information Processing Systems. 2017:6629-6640.

[1]	FU Hong-lin, ZHANG Tai-hong, YANG Ya-ting, Aizimaiti Aiwanier, MA Bo. Scenes Text Modification Network for Uyghur Based on Generative Adversarial Network [J]. Computer and Modernization, 2024, 0(01): 41-46.
[2]	HU Chong-jia, LIU Jin-zhou, FANG Li. Unsupervised Domain Adaptation for Outdoor Point Cloud Semantic Segmentation [J]. Computer and Modernization, 2024, 0(01): 74-79.
[3]	LIN Wei. Incremental News Recommendation Method Based on Self-supervised Learning and Data Replay [J]. Computer and Modernization, 2023, 0(12): 1-6.
[4]	LIANG Tian-kai, HUANG Kang-hua, LIU Kai-hang, LAN Lan, ZENG Bi. Deep Federated Image Classification Method Based on Bilateral Homomorphic Encryption [J]. Computer and Modernization, 2023, 0(12): 36-40.
[5]	QIU Kai-xing, FENG Guang. A Multi-label Image Classification Model Based on Dual Feature Attention [J]. Computer and Modernization, 2023, 0(12): 41-47.
[6]	ZHANG Bo-quan, MAI Hai-peng, CHEN Jia-min, Pang Jin-ju. White Matter Hyperintensities Segmentation Based on High Gray Value#br# Attention Mechanism [J]. Computer and Modernization, 2023, 0(12): 67-75.
[7]	LI Yan-man, WANG Bi-heng, ZHAO Ling-yan. Safety Helmet Detection Based on Lightweight YOLOv5 [J]. Computer and Modernization, 2023, 0(10): 59-64.
[8]	LI Shi-da, XIANG Jian-wen. A Weakened Joint Reinforcement Method to Improve Robustness of Image Recognition Models [J]. Computer and Modernization, 2023, 0(10): 70-76.
[9]	WANG Xin, XIAO Tao-rui. GAN-based Adversarial Attacks on Face Recognition [J]. Computer and Modernization, 2023, 0(10): 115-120.
[10]	SHEN Jia-wei, LU Yi-ming, CHEN Xiao-yi, QIAN Mei-ling, LU Wei-zhong, . Review of Research on Human Behavior Detection Methods Based on Deep Learning [J]. Computer and Modernization, 2023, 0(09): 1-9.
[11]	LIU Chan-yi, HUANG Dan, XUE Lin-yan, WANG Tao, ZHU Tao, . COVID-19 X-ray Classification Based on Improved Efficientnet Network [J]. Computer and Modernization, 2023, 0(09): 94-99.
[12]	MA Guo-xiang, YANG Ling-fei, YAN Chuan-bo, ZHANG Zhi-hao, SUN Bing, WANG Xiao-rong. Ultrasonic Image Diagnosis of Hepatic Echinococcosis Based on Deep DenseNet Network [J]. Computer and Modernization, 2023, 0(09): 100-104.
[13]	NONG Hao-cheng, REN De-jun, REN Qiu-lin, LIU Peng-li, HUANG De-cheng. Surface Anomaly Detection Algorithm of Flexible Plastic Packaging Based on Improved ConvNeXt [J]. Computer and Modernization, 2023, 0(08): 12-17.
[14]	OUYANG Fei, WU Xu, XIANG Dong-sheng. Garbage Classification and Detection Method Based on Improved YOLOX [J]. Computer and Modernization, 2023, 0(08): 68-73.
[15]	HU Rui-jie, CHE Dou. Review of Infrared Small Target Detection [J]. Computer and Modernization, 2023, 0(08): 79-86.