基于孪生特征融合网络的自然场景文本图像超分辨率方法

doi:10.3969/j.issn.1006-2475.2025.02.012

摘要/Abstract

摘要： 自然场景文本图像超分辨率方法旨在提高文本图像的分辨率和可读性，进而提升下游高级文本识别任务的性能。现有研究通过引入文本先验信息能够更好地指导超分辨率重建过程。然而，这些方法未能有效利用文本先验信息并将其与图像特征充分融合，从而限制了超分辨率任务的性能。基于此，本文提出一种基于孪生特征融合网络（Twin Feature Fusion Network， TFFN）的自然场景文本图像超分辨率方法，该方法旨在最大化利用来自预训练文本识别器中的文本先验信息，使其专注于文本区域内容的恢复。首先，利用文本识别网络提取文本先验信息；其次，构建一种孪生特征融合模块，该模块采用孪生注意力机制促进图像特征和文本先验信息之间的双向交互，并利用融合模块进一步融合上下文增强的图像特征和文本先验信息；最后，提取序列特征并重建超分辨率图像。实验结果表明，本文方法在TextZoom数据集的不同难度等级中，ASTER、MORAN和CRNN这3个文本识别网络中的识别准确率分别提升了0.22～0.5、0.6～1.1以及0.33～1.1个百分点。

关键词: 图像超分辨率重建, 文本图像；特征融合；自注意力机制；交叉注意力机制

Abstract: The aim of the scene text image super-resolution （STISR） method is to enhance the resolution and legibility of text images， thereby improving the performance of downstream text recognition tasks. Previous studies have shown that the introduction of text-prior information can better guide the super-resolution. However， these methods have not effectively utilized text-prior information and have not fully integrated it with image features， limiting super-resolution task performance. In this paper， we propose a Twin Feature Fusion Network （TFFN） to address this problem. The method aims to maximize the utilization of text-prior information from pre-trained text recognizers， with a focus on the recovery of text area content. Firstly， text-prior information is extracted using a text recognition network. Next， a twin feature fusion module is constructed， which employs a twin attention mechanism to facilitate bidirectional interaction between image features and text-prior information. The fusion module further integrates context-enhanced image features and text-prior information. Finally， sequence features are extracted to reconstruct the text image. Experiments on the benchmark TextZoom dataset show that the proposed TFFN improves the recognition accuracy of the ASTER， MORAN， and CRNN text recognition networks by 0.22～0.5， 0.6～1.1 and 0.33～1.1 percentage points， respectively.

Key words: super-resolution reconstruction, text images, feature fusion, self-attention mechanism, cross-attention mechanism

中图分类号:

TP391

冯心洁, 王伟. 基于孪生特征融合网络的自然场景文本图像超分辨率方法[J]. 计算机与现代化, 2025, 0(02): 86-93.

FENG Xinjie, WANG Wei. Twin Feature Fusion Network for Scene Text Image Super Resolution[J]. Computer and Modernization, 2025, 0(02): 86-93.

参考文献

［1］ RONEN R， TSIPER S， ANSCHEL O， et al. Glass: Global to local attention for scene-text spotting［C］// Proceedings of the 17th European Conference on Computer Vision. Springer， 2022:249-266.
［2］ TAN Y L， KONG A W K， KIM J J. Pure transformer with integrated experts for scene text recognition［C］// Proceedings of the 17th European Conference on Computer Vision. Springer， 2022:481-497.
［3］ ZHANG Y L， LI K P， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 15th European Conference on Computer Vision （ECCV）. Springer， 2018:294-310.
［4］ LEDIF C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2017:105-114.
［5］邢世帅，刘丹凤，王立国，等．基于空间注意力残差网络的图像超分辨率重建模型［J］．计算机与现代化， 2023（10）：45-52．
［6］陈彤，周登文．基于多级Transformer的超大倍率重建网络：参考图像超分辨率［J］．计算机与现代化， 2022（8）：121-126．
［7］ ZAMIR S W， ARORA A， KHAN S， et al. Restormer: Efficient transformer for high-resolution image restoration［C］// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2022:5718-5729.
［8］ WANG W J， XIE E Z， SUN P Z， et al. TextSR: Content-aware text super-resolution guided by recognition［J］. arXiv preprint arXiv:1909.07113， 2019.
［9］ WANG W J， XIE E Z， LIU X B， et al. Scene text image super-resolution in the wild［C］// Proceedings of the 16th European Conference on Computer Vision. Springer， 2020:650-666.
［10］ MA J Q， GUO S， ZHANG L. Text prior guided scene text image super-resolution［J］. IEEE Transactions on Image Processing， 2023，32:1341-1353.
［11］ MA J Q， LIANG Z T， ZHANG L. A text attention network for spatial deformation robust scene text image super-resolution［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2022:5901-5910.
［12］ GUO H， DAI T， MENG G H， et al. Towards robust scene text image super-resolution via explicit location enhancement［J］. arXiv preprint arXiv:2307.09749， 2023.
［13］ GHEINI M， REN X， MAY J. Cross-attention is all you need: Adapting pretrained transformers for machine translation［J］. arXiv preprint arXiv:2104.08771， 2021.
［14］ JIAN L H， XIONG S L， YAN H， et al. Rethinking cross-attention for infrared and visible image fusion［J］. arXiv preprint arXiv:2401.11675， 2024.
［15］ LIU Y F， ZHANG Y F， GUO Q， et al. Image interpolation based on weighted and blended rational function［C］// Asian Conference on Computer Vision（ACCV 2014 Workshops）. Springer， 2015:78-88.
［16］ TAI Y W， LIU S C， BROWN M S， et al. Super resolution using edge prior and single image detail synthesis［C］// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE， 2010:2400-2407.
［17］ ZHANG K B， LUO S， LI M， et al. Learning stacking regressors for single image super-resolution［J］. Applied Intelligence， 2020，50（12）:4325-4341.
［18］ DONG C， LOY C C， HE K M， et al. Learning a deep convolutional network for image super-resolution［C］ // Proceedings of the 13th European Conference on Computer Vision. Springer， 2014:184-199.
［19］ KIM J， LEE J K， LEE K M. Accurate image super-resolution using very deep convolutional networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016:1646-1654.
［20］ AHN N， KANG B， SOHN K A. Fast， accurate， and lightweight super-resolution with cascading residual network［C］// Proceedings of the 15th European Conference on Computer Vision （ECCV）. Springer， 2018:256-272.
［21］ LAI W S， HUANG J B， AHUJA N， et al. Deep laplacian pyramid networks for fast and accurate super-resolution［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2017:5835-5843.
［22］ TONG T， LI G， LIU X J， et al. Image super-resolution using dense skip connections［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. IEEE， 2017:4809-4817.
［23］ ZHAO C R， FENG S Y， ZHAO B N， et al. Scene text image super-resolution via parallelly contextual attention network［C］// Proceedings of the 29th ACM International Conference on Multimedia. ACM， 2021:2908-2917.
［24］ CHEN J Y， LI B， XUE X Y. Scene text telescope: Text-focused scene image super-resolution［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2021:12021-12030.
［25］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. ACM， 2017:6000-6010.
［26］ CHEN J Y， YU H Y， MA J Q， et al. Text gestalt: Stroke-aware scene text image super-resolution［C］// Proceedings of the 36th AAAI Conference on Artificial Intelligence. AAAI， 2022，36（1）:285-293.
［27］ ZHAO M Y， WANG M， BAI F， et al. C3-STISR: Scene text image super-resolution with triple clues［J］. arXiv preprint arXiv:2204.14044， 2022.
［28］ SHI B G， BAI X， YAO C. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017，39（11）:2298-2304.
［29］ JADERBERG M， SIMONYAN K， ZISSERMAN A， et al. Spatial transformer networks［C］// Proceedings of the 29th International Conference on Neural Information Processing Systems. ACM， 2015，2:2017-2025.
［30］ DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words: Transformers for image recognition at scale［J］. arXiv preprint arXiv:2010.11929， 2020.
［31］ SHI W Z， CABALLERO J， HUSZÁR F， et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE， 2016:1874-1883.
［32］ SHI B G， YANG M K， WANG X G， et al. Aster: An attentional scene text recognizer with flexible rectification［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2019，41（9）:2035-2048.
［33］ LUO C J， JIN L W， SUN Z H. MORAN: A multi-object rectified attention network for scene text recognition［J］. Pattern Recognition， 2019，90:109-118.
［34］ CAI J R， ZENG H， YONG H W， et al. Toward real-world single image super-resolution: A new benchmark and a new model［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. IEEE， 2019:3086-3095.
［35］ ZHANG X E， CHEN Q F， NG R， et al. Zoom to learn， learn to zoom［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE， 2019:3757-3765.
［36］ ZHANG Y L， TIAN Y P， KONG Y， et al. Residual dense network for image restoration［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2021，43（7）:2480-2495.

[1]	王乐, 王志英. 任务中断对用户识别网络钓鱼的漂移扩散实证研究[J]. 计算机与现代化, 2025, 0(02): 1-12.
[2]	封祖焱, 魏延, 陈家坤, 余昕. 基于门控融合的实时语义分割[J]. 计算机与现代化, 2025, 0(02): 121-126.
[3]	韩霄龙, 曾曦, 刘锟, 尚钰. 基于LoRA高效微调通用语言大模型的文本立场检测[J]. 计算机与现代化, 2025, 0(01): 1-6.
[4]	徐胜超, 陈富强 . 基于BP神经网络的食品安全风险预警方法[J]. 计算机与现代化, 2025, 0(01): 20-24.
[5]	陈思贇1, 马怀波2, 张华君2, 兰子柠2, 陈文鑫2, 胡杰1, 常胜1. 基于国产AI芯片的目标检测算法优化与部署[J]. 计算机与现代化, 2025, 0(01): 25-29.
[6]	袁杰, 朱焱 . 基于属性异质图的多目标对抗跨领域推荐[J]. 计算机与现代化, 2025, 0(01): 37-43.
[7]	闫晓奇, 彭逸清, 任小玲. 位置自适应卷积PointNet++的点云数据分类方法[J]. 计算机与现代化, 2025, 0(01): 44-49.
[8]	刘海涛, 冯帆. 基于BD格缩减辅助的连续干扰消除检测算法[J]. 计算机与现代化, 2025, 0(01): 67-73.
[9]	郑久超, 赵新元. 基于主题与描述信息的实体链接方法[J]. 计算机与现代化, 2024, 0(12): 10-14.
[10]	张昆1, 张永伟1, 吴永城1, 张笑文2, 翟世臣2. 基于大模型的设备故障知识图谱自动构建方法[J]. 计算机与现代化, 2024, 0(11): 46-53.
[11]	张宇1, 2, 黎靖1, 2, 马铭1, 2, 王众祥1, 2, 孙妍1, 2. YOLOLW:一个新的轻量级目标检测模型[J]. 计算机与现代化, 2024, 0(11): 91-98.
[12]	杜猛俊1, 李昂1, 童俊1, 钱锦1, 康恺1, 王若丁1, 靳文星2. 基于改进极限学习算法的电力信息数据融合模型[J]. 计算机与现代化, 2024, 0(10): 61-64.
[13]	焦一凯1, 2, 朱欣娟1, 2. 公共文化资源标签推荐方法[J]. 计算机与现代化, 2024, 0(10): 107-112.
[14]	杨俞沣1, 2, 夏小云2, 陈泽丰3, 廖伟志2, 李积武2. 融合多策略蜣螂优化算法的外卖订单配送路径优化[J]. 计算机与现代化, 2024, 0(09): 25-32.
[15]	马钰, 杨勇, 任鸽, 帕力旦·吐尔逊. 基于GCN和微调BERT的作文自动评分方法[J]. 计算机与现代化, 2024, 0(09): 33-37.