Multistage-transformer Large-factor Network: Reference-based Super-resolution

Abstract

Abstract: Image super-resolution （SR） refers to reconstructing the corresponding high-resolution copy from a low-resolution image. Aiming at solving the problem of inaccurate reconstruction of SR in the cases of super-large magnification （8×, 16×）, a multi-level transformer super-magnification reconstruction network is proposed （MTLF）. MTLF performs multi-level stacking of multiple transformers to process features of different scales, and uses the attention weights, which are obtained by the transformer and then improved by the modified attention module, to synthesize finer textures. In the end, the features of all magnifications fuse into a super-large-scale SR image. Experiments resenlts show that MTLF is superior to the state-of-the-art methods （including single-image super-resolution and Ref-based super-resolution methods） in terms of peak signal-to-noise ratio and visual effects. In particular, MTLF achieves fairly good results in the ultimate magnification （32×） scenario.

Key words: reference-based, super-resolution, transformer, large-factor, attention

CHEN Tong, ZHOU Deng-wen. Multistage-transformer Large-factor Network: Reference-based Super-resolution[J]. Computer and Modernization, 2022, 0(08): 121-126.

References

［1］ FREEMAN W T, PASZTOR C E, CARMICHAEL O T. Learning low-level vision［J］. International Journal of Computer Vision, 2000,40:25-47.
［2］ IRANI M, PELEG S. Improving resolution by image registration［J］. CVGIP: Graphical Models and Image Processing, 1991,53（3）:231-239.
［3］ YANG F, YANG H, FU J, et al. Learning texture transformer network for image super-resolution［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:5791-5800.
［4］ ZHANG Y, ZHANG Z, DIVERDI S, et al. Texture hallucination for large-factor painting super-resolution［C］// European Conference on Computer Vision. 2020:209-225.
［5］ DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015,38（2）:295-307.
［6］周登文,赵丽娟,段然,等. 基于递归残差网络的图像超分辨率重建［J］. 自动化学报, 2019,45（6）:1157-1165.
［7］段然,周登文,赵丽娟,等. 基于多尺度特征映射网络的图像超分辨率重建［J］. 浙江大学学报（工学版）, 2019,53（7）:1331-1339.
［8］周登文,马路遥,田金月,等. 基于特征融合注意网络的图像超分辨率重建［J/OL］, 自动化学报, 2021. https://doi.org/10.16383/j.aas.c190428.
［9］陈一鸣,周登文. 基于自适应级联的注意力网络的超分辨重建［J/OL］. 自动化学报, 2021. https://doi.org/10.16383/j.aas.c200035.
［10］李金新,黄志勇,李文斌,等. 基于多层次特征融合的图像超分辨率重建［J/OL］. 自动化学报， 2020. https://doi.org/10.16383/j.aas.c200585.
［11］LAI W S, HUANG J B, AHUJA N, et al. Deep laplacian pyramid networks for fast and accurate super-resolution［C］// 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2017:5835-5843.
［12］ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the European Conference on Computer Vision （ECCV）. 2018:286-301.
［13］ZHANG Z, WANG Z, LIN Z, et al. Image super-resolution by neural texture transfer［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2019:7982-7991.
［14］YAN X, ZHAO W, YUAN K, et al. Towards content-independent multi-reference super-resolution: Adaptive pattern matching and feature aggregation［C］// European Conference on Computer Vision. 2020:52-68.
［15］XIE Y, XIAO J, SUN M, et al. Feature representation matters: End-to-end learning for reference-based image super-resolution［C］// European Conference on Computer Vision. 2020:230-245.
［16］SHIM G, PARK J, KWEON I S. Robust reference-based super-resolution with similarity-aware deformable convolution［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2020:8425-8434.
［17］PARMAR N, VASWANI A, USZKOREIT J, et al. Image transformer［C］// Proceedings of the 35th International Conference on Machine Learning. 2018:4055-4064.
［18］ZENG Y, FU J, CHAO H, et al. Learning pyramid-context encoder network for high-quality image inpainting［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. 2019:1486-1494.
［19］TONG T, LI G, LIU X, et al. Image super-resolution using dense skip connections［C］// Proceedings of the IEEE International Conference on Computer Vision （ICCV）. 2017:4799-4807.
［20］YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions［J］. arXiv preprint arXiv:1511.07122, 2015.
［21］CCIR. Studio encoding parameters of digital television for standard 4〖DK〗∶3 and wide-screen 16〖DK〗∶ 9 aspect ratios：ITU-R BT.601-5［S］. Radio Consultative Committee Int. Telecommun. 2011.
［22］POYNTON C. YUV and luminance considered harmful: A plea for precise terminology in video［J］. Charles Poynton, 2001.
［23］HOR A, ZIOU D. Image quality metrics: PSNR vs. SSIM［C］// 2010 20th International Conference on Pattern Recognition. 2010:2366-2369.
［24］WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity［J］. IEEE Transactions on Image Processing, 2004,13（4）: 600-612.
［25］KINGMA D P, BA J. Adam: A method for stochastic optimization［J］. arXiv preprint arXiv:1412.6980, 2014.
［26］LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition （CVPR） Workshops. 2017:136-144.
［27］LEDIG C, THEIS L, HUSZAR F, et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. 2017:4681-4690.

[1]	HE Sida, CHEN Pinghua. Intent-based Lightweight Self-Attention Network for Sequential Recommendation [J]. Computer and Modernization, 2024, 0(12): 1-9.
[2]	ZHAO Chenyang, XUE Tao, LIU Junhua. Fashion Clothing Pattern Generation Based on Improved Stable Diffusion [J]. Computer and Modernization, 2024, 0(12): 15-23.
[3]	HUANG Tingpei1, MA Lubiao1, LI Shibao2, LIU Jianhang1. Gesture Recognition Method Based on WiFi and Prototypical Network [J]. Computer and Modernization, 2024, 0(12): 34-39.
[4]	ZHANG Xiaodong1, BAI Guangzhi1, LI Min1, LI Haoyang2. Oil and Gas Well Production Prediction Model Based on Empirical Wavelet Transform [J]. Computer and Modernization, 2024, 0(12): 53-58.
[5]	WANG Haiyang, GONG Tongxin, YANG Jintao, CHEN Zailong. Short-term Load Forecasting in Industrial Parks with Multi-scale Time Coding [J]. Computer and Modernization, 2024, 0(12): 59-65.
[6]	LIU Yunhai1, Feng Guang1, WU Xiaoting2, YANG Qun2 . Safety Helmet Wearing Detection Algorithm for Complex Construction Scenes [J]. Computer and Modernization, 2024, 0(12): 66-71.
[7]	GU Yue, DENG Songfeng, SHEN Ji, MU Wentao, ZHAO Enqi. SAR Ship Detection Algorithm Based on Improved YOLOv8 [J]. Computer and Modernization, 2024, 0(12): 78-83.
[8]	WANG Yanyuan, MAO Zhengchong. Detection and Recognition Algorithms for Chinese and English Scene Text Images [J]. Computer and Modernization, 2024, 0(12): 84-90.
[9]	LI Junchao1, YOU Fei1, ZHANG Chao2, SU Lele2, GONG Yan2 . BiLSTM-Attention Prediction Model and Error Analysis #br# Based on Novel Multi-objective Coati Optimization Algorithm [J]. Computer and Modernization, 2024, 0(11): 70-76.
[10]	WAN Hongwei, CHEN Pinghua. Polyp Segmentation Based on Involution and Coordinate Reverse Attention [J]. Computer and Modernization, 2024, 0(11): 84-90.
[11]	ZHANG Yu1, 2, LI Jing1, 2, MA Ming1, 2, WANG Zhongxiang1, 2, SUN Yan1, 2. YOLOLW: A Novel Lightweight Object Detection Model [J]. Computer and Modernization, 2024, 0(11): 91-98.
[12]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[13]	YANG Jun1, HU Wei1, ZHU Wenfu2. Visual SLAM Loop Closure Detection Algorithm Based on Improved MobileNetV3 [J]. Computer and Modernization, 2024, 0(10): 21-26.
[14]	WEI Xuecheng1, JIANG Lingyun1, LI Yan2, HE Fei2. Improved Roadside Monocular View Small Target Detection Algorithm Based on YOLOv5 [J]. Computer and Modernization, 2024, 0(10): 27-34.
[15]	DU Mengjun1, LI Ang1, TONG Jun1, QIAN Jin1, KANG Kai1, WANG Ruoding1, JIN Wenxing2. Power Information Data Fusion Model Based on Improved Extreme Learning Algorithm [J]. Computer and Modernization, 2024, 0(10): 61-64.