Twin Feature Fusion Network for Scene Text Image Super Resolution

doi:10.3969/j.issn.1006-2475.2025.02.012

Abstract

Abstract: The aim of the scene text image super-resolution （STISR） method is to enhance the resolution and legibility of text images， thereby improving the performance of downstream text recognition tasks. Previous studies have shown that the introduction of text-prior information can better guide the super-resolution. However， these methods have not effectively utilized text-prior information and have not fully integrated it with image features， limiting super-resolution task performance. In this paper， we propose a Twin Feature Fusion Network （TFFN） to address this problem. The method aims to maximize the utilization of text-prior information from pre-trained text recognizers， with a focus on the recovery of text area content. Firstly， text-prior information is extracted using a text recognition network. Next， a twin feature fusion module is constructed， which employs a twin attention mechanism to facilitate bidirectional interaction between image features and text-prior information. The fusion module further integrates context-enhanced image features and text-prior information. Finally， sequence features are extracted to reconstruct the text image. Experiments on the benchmark TextZoom dataset show that the proposed TFFN improves the recognition accuracy of the ASTER， MORAN， and CRNN text recognition networks by 0.22～0.5， 0.6～1.1 and 0.33～1.1 percentage points， respectively.

Key words: super-resolution reconstruction, text images, feature fusion, self-attention mechanism, cross-attention mechanism

CLC Number:

TP391

FENG Xinjie, WANG Wei. Twin Feature Fusion Network for Scene Text Image Super Resolution[J]. Computer and Modernization, 2025, 0(02): 86-93.

References

［1］ RONEN R， TSIPER S， ANSCHEL O， et al. Glass: Global to local attention for scene-text spotting［C］// Proceedings of the 17th European Conference on Computer Vision. Springer， 2022:249-266.
［2］ TAN Y L， KONG A W K， KIM J J. Pure transformer with integrated experts for scene text recognition［C］// Proceedings of the 17th European Conference on Computer Vision. Springer， 2022:481-497.
［3］ ZHANG Y L， LI K P， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 15th European Conference on Computer Vision （ECCV）. Springer， 2018:294-310.
［4］ LEDIF C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2017:105-114.
［5］邢世帅，刘丹凤，王立国，等．基于空间注意力残差网络的图像超分辨率重建模型［J］．计算机与现代化， 2023（10）：45-52．
［6］陈彤，周登文．基于多级Transformer的超大倍率重建网络：参考图像超分辨率［J］．计算机与现代化， 2022（8）：121-126．
［7］ ZAMIR S W， ARORA A， KHAN S， et al. Restormer: Efficient transformer for high-resolution image restoration［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2022:5718-5729.
［8］ WANG W J， XIE E Z， SUN P Z， et al. TextSR: Content-aware text super-resolution guided by recognition［J］. arXiv preprint arXiv:1909.07113， 2019.
［9］ WANG W J， XIE E Z， LIU X B， et al. Scene text image super-resolution in the wild［C］// Proceedings of the 16th European Conference on Computer Vision. Springer， 2020:650-666.
［10］ MA J Q， GUO S， ZHANG L. Text prior guided scene text image super-resolution［J］. IEEE Transactions on Image Processing， 2023，32:1341-1353.
［11］ MA J Q， LIANG Z T， ZHANG L. A text attention network for spatial deformation robust scene text image super-resolution［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2022:5901-5910.
［12］ GUO H， DAI T， MENG G H， et al. Towards robust scene text image super-resolution via explicit location enhancement［J］. arXiv preprint arXiv:2307.09749， 2023.
［13］ GHEINI M， REN X， MAY J. Cross-attention is all you need: Adapting pretrained transformers for machine translation［J］. arXiv preprint arXiv:2104.08771， 2021.
［14］ JIAN L H， XIONG S L， YAN H， et al. Rethinking cross-attention for infrared and visible image fusion［J］. arXiv preprint arXiv:2401.11675， 2024.
［15］ LIU Y F， ZHANG Y F， GUO Q， et al. Image interpolation based on weighted and blended rational function［C］// Asian Conference on Computer Vision（ACCV 2014 Workshops）. Springer， 2015:78-88.
［16］ TAI Y W， LIU S C， BROWN M S， et al. Super resolution using edge prior and single image detail synthesis［C］// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE， 2010:2400-2407.
［17］ ZHANG K B， LUO S， LI M， et al. Learning stacking regressors for single image super-resolution［J］. Applied Intelligence， 2020，50（12）:4325-4341.
［18］ DONG C， LOY C C， HE K M， et al. Learning a deep convolutional network for image super-resolution［C］ // Proceedings of the 13th European Conference on Computer Vision. Springer， 2014:184-199.
［19］ KIM J， LEE J K， LEE K M. Accurate image super-resolution using very deep convolutional networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016:1646-1654.
［20］ AHN N， KANG B， SOHN K A. Fast， accurate， and lightweight super-resolution with cascading residual network［C］// Proceedings of the 15th European Conference on Computer Vision （ECCV）. Springer， 2018:256-272.
［21］ LAI W S， HUANG J B， AHUJA N， et al. Deep laplacian pyramid networks for fast and accurate super-resolution［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2017:5835-5843.
［22］ TONG T， LI G， LIU X J， et al. Image super-resolution using dense skip connections［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE， 2017:4809-4817.
［23］ ZHAO C R， FENG S Y， ZHAO B N， et al. Scene text image super-resolution via parallelly contextual attention network［C］// Proceedings of the 29th ACM International Conference on Multimedia. ACM， 2021:2908-2917.
［24］ CHEN J Y， LI B， XUE X Y. Scene text telescope: Text-focused scene image super-resolution［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021:12021-12030.
［25］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM， 2017:6000-6010.
［26］ CHEN J Y， YU H Y， MA J Q， et al. Text gestalt: Stroke-aware scene text image super-resolution［C］// Proceedings of the 36th AAAI Conference on Artificial Intelligence. AAAI， 2022，36（1）:285-293.
［27］ ZHAO M Y， WANG M， BAI F， et al. C3-STISR: Scene text image super-resolution with triple clues［J］. arXiv preprint arXiv:2204.14044， 2022.
［28］ SHI B G， BAI X， YAO C. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017，39（11）:2298-2304.
［29］ JADERBERG M， SIMONYAN K， ZISSERMAN A， et al. Spatial transformer networks［C］// Proceedings of the 29th International Conference on Neural Information Processing Systems. ACM， 2015，2:2017-2025.
［30］ DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words: Transformers for image recognition at scale［J］. arXiv preprint arXiv:2010.11929， 2020.
［31］ SHI W Z， CABALLERO J， HUSZÁR F， et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016:1874-1883.
［32］ SHI B G， YANG M K， WANG X G， et al. Aster: An attentional scene text recognizer with flexible rectification［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2019，41（9）:2035-2048.
［33］ LUO C J， JIN L W， SUN Z H. MORAN: A multi-object rectified attention network for scene text recognition［J］. Pattern Recognition， 2019，90:109-118.
［34］ CAI J R， ZENG H， YONG H W， et al. Toward real-world single image super-resolution: A new benchmark and a new model［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. IEEE， 2019:3086-3095.
［35］ ZHANG X E， CHEN Q F， NG R， et al. Zoom to learn， learn to zoom［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2019:3757-3765.
［36］ ZHANG Y L， TIAN Y P， KONG Y， et al. Residual dense network for image restoration［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2021，43（7）:2480-2495.

[1]	GUO Hua. Resampling of Imbalanced Data for Optimizing Downstream Tasks [J]. Computer and Modernization, 2025, 0(02): 28-32.
[2]	LIU Chongyi, LI Hua, REN Dejun, LIU Yaokai, WANG Yulong. Anomaly Detection Algorithm Based on Bidirectional Multi-scale Knowledge Distillation [J]. Computer and Modernization, 2025, 0(02): 58-63.
[3]	ZHAO Yin, YIN Siqing, ZHANG Yonglai. Improved Traffic Sign Detection Algorithm of YOLOv7 [J]. Computer and Modernization, 2025, 0(02): 94-99.
[4]	FENG Zuyan, WEI Yan, CHEN Jiakun, YU Xin. Real-time Semantic Segmentation Based on Gate-controlled Fusion [J]. Computer and Modernization, 2025, 0(02): 121-126.
[5]	HE Sida, CHEN Pinghua. Intent-based Lightweight Self-Attention Network for Sequential Recommendation [J]. Computer and Modernization, 2024, 0(12): 1-9.
[6]	ZHANG Simin, LIU Xinmei, YIN Junling, LI Baoling. PCB Defect Detection Method Based on Improved YOLOv7 [J]. Computer and Modernization, 2024, 0(12): 45-52.
[7]	ZHANG Xiaodong1, BAI Guangzhi1, LI Min1, LI Haoyang2. Oil and Gas Well Production Prediction Model Based on Empirical Wavelet Transform [J]. Computer and Modernization, 2024, 0(12): 53-58.
[8]	WANG Haiyang, GONG Tongxin, YANG Jintao, CHEN Zailong. Short-term Load Forecasting in Industrial Parks with Multi-scale Time Coding [J]. Computer and Modernization, 2024, 0(12): 59-65.
[9]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[10]	HOU Congying, YANG Wengqing, WANG Zhao, CHENG Cong. Speech Enhancement Based on Time-frequency Self-attention Residual Temporal#br# Convolutional Networks [J]. Computer and Modernization, 2024, 0(09): 20-24.
[11]	MA Yu, YANG Yong, REN Ge, Palidan Tuerxun. Automated Essay Scoring Method Based on GCN and Fine Tuned BERT [J]. Computer and Modernization, 2024, 0(09): 33-37.
[12]	ZHENG Shangpo1, CHEN Defu1, LI Jianli2, LIN Guoxian2, WANG Xingping3. Pedestrian Tracking Algorithm Based on Improved YOLOv5s and DeepSORT [J]. Computer and Modernization, 2024, 0(08): 54-58.
[13]	PANG Mei, WANG Gong, ZHAN Yong, HUANG Zhefa. Underwater Trash Detection Method Based on Improved YOLOv5 [J]. Computer and Modernization, 2024, 0(07): 120-126.
[14]	FU Lingli, QIU Yu, ZHANG Xinchen. Retinal Vessel Segmentation Based on Improved U-Net with Multi-feature Fusion [J]. Computer and Modernization, 2024, 0(06): 76-82.
[15]	ZHU Fen, HE Lifeng, SUN Shuang, ZHANG Mengying, YU Jiajia. Pancreas Segmentation Model Based on Deformable Residual and Cascading Encoding [J]. Computer and Modernization, 2024, 0(06): 83-88.