基于可分离结构变换的轻量级Vision Transformer

摘要/Abstract

摘要： 由于视觉Transformer结构模型参数量大、浮点计算次数高，使得其难以部署到终端设备上。因为注意力矩阵存在低秩瓶颈，所以模型压缩算法和注意力机制加速算法不能很好地平衡模型参数量、模型推理速度和模型性能之间的关系。为了解决上述问题，本文设计一种轻量级的ViT-SST模型用于图像分类任务。首先，通过将传统全连接层转换为可分离结构，大幅度降低模型参数量且提高了模型推理速度，保证了注意力矩阵不会因出现低秩而破坏模型表达能力；其次，提出一种基于SVD分解的克罗内克积近似分解法，可以将公开的ViT-Base模型预训练参数转换至ViT-Base-SST模型，略微缓解了ViT模型的过拟合现象并提高了模型精度。在常见公开图片数据集CIFAR系列和Caltech系列上的实验验证了本文方法优于对比方法

关键词: 深度学习, 计算机视觉, 图像分类, 模型压缩

Abstract: Due to a large number of parameters and high floating-point calculations of the Visual Transformer model, it is difficult to deploy it to portable or terminal devices. Because the attention matrix has a low-rank bottleneck, the model compression algorithm and the attention mechanism acceleration algorithm cannot well balance the relationship between the amount of model parameters, model inference speed and model performance. In order to solve the above problems, a lightweight ViT-SST model is designed. Firstly, by transforming the traditional fully connected layer into a separable structure, the number of model parameters is greatly reduced and the reasoning speed of the model is improved, and it is guaranteed that the attention matrix will not destroy the model’s expressive ability due to the appearance of low rank. Secondly, this paper proposes a Kronecker product approximate decomposition method based on SVD decomposition, which can convert the pre-training parameters of the public ViT-Base model to the ViT-Base-SST model. It slightly alleviates the overfitting phenomenon of the ViT-Base model and improves the accuracy of the model. Experiments on five common public datasets show that the proposed method is more suitable for the Transformer structure model than traditional compression methods.

Key words: deep learning, computer vision, image classification, model compression

黄延辉, 兰海, 魏宪. 基于可分离结构变换的轻量级Vision Transformer[J]. 计算机与现代化, 2022, 0(10): 75-81.

HUANG Yan-hui, LAN Hai, WEI Xian. Lightweight Vision Transformer Based on Separable Structured Transformations[J]. Computer and Modernization, 2022, 0(10): 75-81.

参考文献［36］

［1］	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks［C］// Proceedings of the 2012 International Conference on Neural Information Processing Systems. 2012:1097-1105.
［2］	DENG J, DOND W, SOCHER R, et al. ImageNet: A large-scale hierarchical image database［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009:248-255.
［3］	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［C］// Proceedings of the 2015 International Conference on Learning Representations. 2015:212-219.
［4］	HE K M, ZHZANG X Y, REN S Q, et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:770-778.
［5］	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale［C］// Proceedings of the 2020 International Conference on Learning Representations. 2020:321-326.
［6］	RIQUELME C, PUIGCERVER J, MUSTAFA B, et al. Scaling vision with sparse mixture of experts［C］// Proceedings of the 2021 International Conference on Neural Information Processing Systems. 2021:1097-1105.
［7］	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. 2021:9992-10002.
［8］	STRUDEL R, GARCIA R, LAPTEV I, et al. Segmenter: Transformer for semantic segmentation［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. 2021:7242-7252.
［9］	TAY Y, DEHGHANI M, BAHRI D, et al. Efficient transformers: A survey［J］. arXiv preprint arXiv:2009.06732, 2020.
［10］	WANG S N, LI B Z, KHABSA M, et al. Linformer: Self-attention with linear complexity［J］. arXiv preprint arXiv:2006.04768, 2020.
［11］	CHOROMANSKI K, LIKHOSHERSTOV V, DOHAN D, et al. Rethinking attention with performers［C］// Proceedings of the 2021 International Conference on Learning Representations. 2021:3-7.〖HJ0.68mm〗
［12］	KITAEV N, KAISER L, LEVSKAYA A. Reformer: The efficient transformer［C］// Proceedings of the 2020 International Conference on Learning Representations. 2020:1-7.
［13］	CHILD R, GRAY S, RADFORD A, et al. Generating long sequences with sparse transformers［J］. arXiv preprint arXiv:1904.10509, 2019.
［14］	BHOJANAPALLI S, YUN C, RAWAT A S, et al. Low-rank bottleneck in multi-head attention models［C］// Proceedings of the 2020 International Conference on Machine Learning. 2020:864-873.
［15］	YU X Y, LIU T L, WANG X C, et al. On compressing deep models by low rank and sparse decomposition［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017:67-76.
［16］	WU J X, LENG C, WANG Y H, et al. Quantized convolutional neural networks for mobile devices［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:4820-4828.
［17］	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network［J］. arXiv preprint arXiv:1503.02531, 2015.
［18］	HANSON S J, PRATT L Y. Comparing biases for minimal network construction with back-propagation［C］// Proceedings of the 1st International Conference on Neural Information Processing Systems. 1988:177-185.
［19］	WU H P, XIAO B, CODELLA N, et al. CvT: Introducing convolutions to vision transformers［C］// Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. 2021:22-31.
［20］	BELTAGY I, PETERS M E, COHAN A. Longformer: The long-document transformer［J］. arXiv preprint arXiv:2004.05150, 2020.
［21］	TAY Y, BAHRI D, METZLER D, et al. Synthesizer: Rethinking self-attention for transformer models［C］// Proceedings of the 38th IEEE International Conference on Machine Learning. 2021:10183-10192.
［22］	DAI Z H, YANG Z L, YANG Y M, et al. Transformer-XL: Attentive language models beyond a fixed-length context［J］. arXiv preprint arXiv:1901.02860, 2019.
［23］	RAE J W, POTAPENKO A, JAYAKUMAR S M, et al. Compressive transformers for long-range sequence modelling［C］// Proceedings of the 2020 International Conference on Learning Representations. 2020:26-30.
［24］	TAI C, XIAO T, ZHANG Y, et al. Convolutional neural networks with low-rank regularization［J］. arXiv preprint arXiv:1511.06067, 2015.
［25］	GONG Y C, LIU L, YANG M, et al. Compressing deep convolutional networks using vector quantization［J］. arXiv preprint arXiv:1412.6115, 2014.
［26］	VANHOUCKE V, SENIOR A, MAO M Z. Improving the speed of neural networks on CPUs［EB/OL］. (2011-12-15)［2021-10-02］. https://research.google/pubs/pub37631/.
［27］	ROMERO A, BALLAS N, KAHOU S E, et al. FitNets: Hints for thin deep nets［J］. arXiv preprint arXiv:1412.6550, 2014.
［28］	MOZER M C, JORDAN M I, PETSCHE T. Advances in Neural Information Processing Systems 9［M］. Morgan Kaufmann Publishers, 1997.
［29］	HASSIBI B, STORK D G. Second order derivatives for network pruning: Optimal brain surgeon［C］// Proceedings of the 1992 International Conference on Neural Information Processing Systems. 1992:164-171.
［30］	SRINIVAS S, BABU R V. Data-free parameter pruning for deep neural networks［C］// Proceedings of the 2015 British Machine Vision Conference. 2015. DOI: 10.5244/C.29.31.
［31］	DENG L. The MNIST database of handwritten digit images for machine learning research ［best of the Web］［J］. IEEE Signal Processing Magazine, 2012,29(6):141-142.
［32］	KRIZHEVSKY A. Learning multiple layers of features from tiny images［R］. University of Toronto, 2009.
［33］	GRIFFIN G, HOLUB A, PERONA P. Caltech-256 object category dataset［EB/OL］. (2007-12-15)［2021-12-15］. https://authors.library.caltech.edu/7694/?〖KG-*3〗ref=https://githubhelp.com.
［34］	SEHWAG V, WANG S Q, MITTAL P, et al. HYDRA: Pruning adversarially robust neural networks［C］// Proceedings of the 2020 International Conference on Neural Information Processing Systems. 2020:97-105.
［35］	LIU N, MA X L, XU Z Y, et al. AutoCompress: An automatic DNN structured pruning framework for ultra-high compression rates［C］// Proceedings of the 2020 AAAI Conference on Artificial Intelligence. 2020:4876-4883.
［36］	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. 2017:618-626.

[1]	胡崇佳, 刘金洲, 方立. 基于无监督域适应的室外点云语义分割[J]. 计算机与现代化, 2024, 0(01): 74-79.
[2]	林威. 基于自监督学习和数据回放的新闻推荐模型增量学习方法[J]. 计算机与现代化, 2023, 0(12): 1-6.
[3]	梁天恺, 黄康华, 刘凯航, 兰岚, 曾碧. 基于双向同态加密的深度联邦图片分类方法[J]. 计算机与现代化, 2023, 0(12): 36-40.
[4]	邱凯星, 冯广. 基于双重特征注意力的多标签图像分类模型[J]. 计算机与现代化, 2023, 0(12): 41-47.
[5]	张伯泉, 麦海鹏, 陈嘉敏, 逄锦聚. 基于高灰度值注意力机制的脑白质高信号分割[J]. 计算机与现代化, 2023, 0(12): 67-75.
[6]	马泽宇, 叶宁, 徐康, 王甦, 王汝传, . 基于FMCW雷达和ResNeSt-GRU的行为识别方法[J]. 计算机与现代化, 2023, 0(11): 101-107.
[7]	李延满, 王必恒, 赵羚焱. 基于轻量化YOLOv5的安全帽检测[J]. 计算机与现代化, 2023, 0(10): 59-64.
[8]	黎世达, 项剑文. 一种提高图像识别模型鲁棒性的弱化强化方法[J]. 计算机与现代化, 2023, 0(10): 70-76.
[9]	沈加炜, 陆一鸣, 陈晓艺, 钱美玲, 陆卫忠, . 基于深度学习的人体行为检测方法研究综述[J]. 计算机与现代化, 2023, 0(09): 1-9.
[10]	吴甜, 刘海华, 童顺延. 基于深度反馈的卷积神经网络的图像分类[J]. 计算机与现代化, 2023, 0(09): 82-86.
[11]	刘禅奕, 黄丹, 薛林雁, 王涛, 朱桃, . 改进EfficientNet网络的COVID-19 X光分类[J]. 计算机与现代化, 2023, 0(09): 94-99.
[12]	马国祥, 杨凌菲, 严传波, 张志豪, 孙彬, 王晓荣. 基于深度DenseNet网络的肝包虫病超声影像诊断方法[J]. 计算机与现代化, 2023, 0(09): 100-104.
[13]	农皓程, 任德均, 任秋霖, 刘澎笠, 黄德成. 基于改进ConvNeXt的软塑包装表面异常检测算法[J]. 计算机与现代化, 2023, 0(08): 12-17.
[14]	欧阳飞, 吴旭, 向东升. 基于改进YOLOX的垃圾分类检测方法[J]. 计算机与现代化, 2023, 0(08): 68-73.
[15]	胡睿杰, 车逗. 红外小目标检测方法综述[J]. 计算机与现代化, 2023, 0(08): 79-86.