降低参数规模的卷积神经网络模型压缩方法

计算机与现代化 ›› 2021, Vol. 0 ›› Issue (09): 83-89.

降低参数规模的卷积神经网络模型压缩方法

(1. Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China;
2. University of Chinese Academy of Sciences, Beijing 100049, China)

出版日期:2021-09-14 发布日期:2021-09-14
作者简介:朱雪晨(1996—),女,河南新乡人,硕士研究生,研究方向:FPGA神经网络加速器设计,E-mail: 18703862561@163.com; 陈三林(1996—),男,硕士研究生,研究方向:数字集成电路设计,E-mail: chensanlin19@mails.ucas.ac.cn; 通信作者:蔡刚(1980—),男,高级工程师,博士,研究方向:大规模集成电路设计,人工智能,E-mail: caig@mail.ie.ac.cn; 黄志洪(1984—),男,高级工程师,博士,研究方向:可编程芯片设计技术,FPGA神经网络加速器设计,E-mail: huangzhihong@mail.ie.ac.cn。
基金资助:
国家自然科学基金资助项目(61704173)

Compression Method of CNN Model for Parameter Reduction

(1.中国科学院空天信息创新研究院,北京100094;2.中国科学院大学,北京100049)

Online:2021-09-14 Published:2021-09-14

摘要/Abstract

摘要： 针对卷积神经网络模型参数规模越来越大导致难以在计算与存储资源有限的嵌入式设备上大规模部署的问题，提出一种降低参数规模的卷积神经网络模型压缩方法。通过分析发现，卷积层参数量与输入输出特征图数量以及卷积核大小有关，而全连接层参数数量众多且难以大幅减少。通过分组卷积减少输入输出特征图数量，通过卷积拆分减小卷积核大小，同时采用全局平均池化层代替全连接层的方法来解决全连接层参数数量众多的问题。将上述方法应用于LeNet5和AlexNet进行实验，实验结果表明通过使用组合压缩方法对LeNet5模型进行最大压缩后，参数规模可减少97%，识别准确率降低了不到2个百分点，而压缩后的AlexNet模型参数规模可减少95%，识别准确率提高了6.72个百分点，在保证卷积神经网络精度的前提下，可大幅减少模型的参数量。

关键词: 卷积神经网络, 参数规模, 分组卷积, 卷积拆分, 全局平均池化

Abstract: In order to solve the problem that it is difficult to deploy convolutional neural network model on embedded devices with limited computing and storage resources due to the increasing scale of parameters, a convolutional neural network model compression method is proposed to reduce the scale of parameters. It is found that the number of convolution layer parameters is related to the number of input and output feature maps and the size of convolution kernel, while the number of full connection layer parameters is large and difficult to be reduced significantly. The number of input and output feature maps is reduced by grouping convolution, and the convolution kernel size is reduced by convolution resolution. At the same time, the global average pooling layers are used to replace the fully connected layers to solve the problem of large number of parameters in the fully connected layers. The above methods are applied to LeNet5 and AlexNet for experiments, the experimental results show that the parameters of LeNet5 model can be reduced by 97% and the recognition accuracy can be reduced by less than 2 percentage points by using the combined compression method, the parameters of AlexNet model can be reduced by 95% and the recognition accuracy can be improved by 6.72 percentage points after compression. On the premise of ensuring the accuracy of convolutional neural network, the parameters of the model can be greatly reduced.

Key words: convolutional neural networks, parameter scale, grouping convolution, convolution resolution, global average pooling

朱雪晨, 陈三林, 蔡刚, 黄志洪. 降低参数规模的卷积神经网络模型压缩方法[J]. 计算机与现代化, 2021, 0(09): 83-89.

ZHU Xue-chen, CHEN San-lin, CAI Gang, HUANG Zhi-hong. Compression Method of CNN Model for Parameter Reduction[J]. Computer and Modernization, 2021, 0(09): 83-89.

参考文献［22］

［1］	BALOG M, GAUNT A L, BROCKSCHMIDT M, et al. Deepcoder: Learning to write programs［J］. Machine Learning, 2017， arXiv:1611.01989.
［2］	ERHAN D, SZEGEDY C, TOSHEV A, et al. Scalable object detection using deep neural networks［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014:2155-2162.
［3］	NAKJAI P, KATANYUKUL T. Hand sign recognition for Thai finger spelling: An application of convolution neural network［J］. Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology, 2019,91(2):131-146.
［4］	朱亮. 卷积神经网络的全可编程SOC实现［D］. 昆明：云南大学, 2017.
［5］	邹蕾,张先锋. 人工智能及其发展应用［J］. 信息网络安全, 2012(2):11-3.
［6］	HAILESELLASIE M, HASAN S R, KHALID F, et al. FPGA-based convolutional neural network architecture with reduced parameter requirements［C］// 2018 IEEE International Symposium on Circuits and Systems. 2018:1-5.
［7］	CHENG Y, WANG D, ZHOU P, et al. Model compression 〖HJ1.08mm〗and acceleration for deep neural networks: The principles, progress, and challenges［J］. IEEE Signal Processing Magazine, 2018,35(1):126-136.
［8］	杨扬,蓝章礼,陈巍. 基于统计分析的卷积神经网络模型压缩方法［J］. 计算机系统应用, 2018,27(8):53-59.
［9］	邹月娴,余嘉胜,陈泽晗,等. 图像分类卷积神经网络的特征选择模型压缩方法［J］. 控制理论与应用, 2017,34(6):746-752.
［10］	黄芬芬. 深度卷积神经网络模型压缩方法研究及应用［D］. 北京:北京邮电大学, 2018.
［11］	DENTON E L, ZAREMBA W, BRUNA J, et al. Exploiting linear structure within convolutional networks for efficient evaluation［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014:1269-1277.
［12］	JIAO L, LUO C, CAO W, et al. Accelerating low bit-width convolutional neural networks with embedded FPGA［C］// 2017 27th International Conference on Field Programmable Logic and Applications. 2017:1-4.
［13］	SAHA S, VARMA G, JAWAHAR C V. Compressing deep neural networks for recognizing places［C］// 2017 4th IAPR Asian Conference on Pattern Recognition. 2017:352-357.
［14］	HAN S, MAO H, DALLY W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding［J］. Computer Vision and Pattern Recognition, 2016， arXiv:1510.00149.
［15］	WEI X, CHEN H, LIU W C, et al. Mixed-precision quantization for CNN-based remote sensing scene classification［J］. IEEE Geoscience and Remote Sensing Letters, 2020(99):1-5.
［16］	QIU J T, WANG J, SONG Y, et al. Going deeper with embedded FPGA platform for convolutional neural network［C］// Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 2016:26-35.
［17］	ARKAH Z M, ALZUBAIDI L S. Convolutional neural network with global average pooling for image classification［C］// International Conference on Electrical, Communication, Electronics, Instrumentation and Computing. 2020:171-180.
［18］	PANG Y W, SUN M L, JIANG X C, et al. Convolution in convolution for network in network［J］. IEEE Transactions on Neural Networks and Learning Systems, 2016(99):1587-1597.
［19］	SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015:1-9.
［20］	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:770-778.
［21］	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks［J］. Communications of the ACM, 2017,60(6):84-90.
［22］	YANG Z J, LEI WANG L, LUO L, et al. Bactran: A hardware batch normalization implementation for CNN training engine［J］. IEEE Embedded Systems Letters, 2021,13(1):29-32.

[1]	周成诚, 曾庆军, 杨康, 胡家铭, 韩春伟. 基于高效通道注意力模块的运动想象脑电识别[J]. 计算机与现代化, 2023, 0(12): 19-23.
[2]	刘付琪, 张达, 宋建华, 王海东. 基于CNN-BiLSTM的液压系统故障诊断[J]. 计算机与现代化, 2023, 0(09): 10-19.
[3]	吴甜, 刘海华, 童顺延. 基于深度反馈的卷积神经网络的图像分类[J]. 计算机与现代化, 2023, 0(09): 82-86.
[4]	孙子雨, 任燃, 魏曦哲. 基于DTW-TCN的股票分类及预测研究[J]. 计算机与现代化, 2023, 0(08): 31-37.
[5]	江蕾, 唐建, 杨超越, 吕婷婷. 基于CWGAN-GP与CNN的轴承故障诊断方法[J]. 计算机与现代化, 2023, 0(07): 1-6.
[6]	许叶彤, 耿信哲, 赵伟强, 张月, 宁海龙, 雷涛. 基于CNN-Transformer混合结构的遥感影像变化检测模型[J]. 计算机与现代化, 2023, 0(07): 79-85.
[7]	朱剑波, 葛明锋, 董文飞. 基于改进EfficientNet的阿尔兹海默症图像分类[J]. 计算机与现代化, 2023, 0(06): 56-61.
[8]	刘甲甲, 胡旭欣, 余萍. 聚合多维注意力特征的单目深度估计方法[J]. 计算机与现代化, 2023, 0(06): 76-81.
[9]	刘静, 陈金广. 基于通道注意力和Transformer的图像标题生成方法[J]. 计算机与现代化, 2023, 0(05): 8-12.
[10]	王欣怡, 尹四清, 洪军. 融合注意力机制的非对称深度监督哈希[J]. 计算机与现代化, 2023, 0(05): 26-31.
[11]	苏金库, 桂智明. 基于时空特征的短时出租车流量预测[J]. 计算机与现代化, 2023, 0(05): 32-38.
[12]	王娟, 李传庚, 张卿源, 夏承遗. 基于Multiscale-Net的膝关节半月板分割方法[J]. 计算机与现代化, 2023, 0(05): 111-116.
[13]	王磊, 张晓东, 戴欢. 基于1D-CNN-LSTM注意力网络的抽油机井故障诊断[J]. 计算机与现代化, 2023, 0(04): 1-6.
[14]	徐涯昕, 何泽恩, 徐绪堪. 基于CNN-BiLSTM网络的数控机床故障文本自动分类[J]. 计算机与现代化, 2023, 0(04): 7-14.
[15]	朱原冶, 倪建军, 唐广翼. 一种基于改进卷积神经网络的RGB-D室内场景分类方法[J]. 计算机与现代化, 2023, 0(04): 73-77.