降低参数规模的卷积神经网络模型压缩方法

计算机与现代化 ›› 2021, Vol. 0 ›› Issue (09): 83-89.

降低参数规模的卷积神经网络模型压缩方法

(1. Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China;
2. University of Chinese Academy of Sciences, Beijing 100049, China)

出版日期:2021-09-14 发布日期:2021-09-14
作者简介:朱雪晨(1996—),女,河南新乡人,硕士研究生,研究方向:FPGA神经网络加速器设计,E-mail: 18703862561@163.com; 陈三林(1996—),男,硕士研究生,研究方向:数字集成电路设计,E-mail: chensanlin19@mails.ucas.ac.cn; 通信作者:蔡刚(1980—),男,高级工程师,博士,研究方向:大规模集成电路设计,人工智能,E-mail: caig@mail.ie.ac.cn; 黄志洪(1984—),男,高级工程师,博士,研究方向:可编程芯片设计技术,FPGA神经网络加速器设计,E-mail: huangzhihong@mail.ie.ac.cn。
基金资助:
国家自然科学基金资助项目(61704173)

Compression Method of CNN Model for Parameter Reduction

(1.中国科学院空天信息创新研究院,北京100094;2.中国科学院大学,北京100049)

Online:2021-09-14 Published:2021-09-14

摘要/Abstract

摘要： 针对卷积神经网络模型参数规模越来越大导致难以在计算与存储资源有限的嵌入式设备上大规模部署的问题，提出一种降低参数规模的卷积神经网络模型压缩方法。通过分析发现，卷积层参数量与输入输出特征图数量以及卷积核大小有关，而全连接层参数数量众多且难以大幅减少。通过分组卷积减少输入输出特征图数量，通过卷积拆分减小卷积核大小，同时采用全局平均池化层代替全连接层的方法来解决全连接层参数数量众多的问题。将上述方法应用于LeNet5和AlexNet进行实验，实验结果表明通过使用组合压缩方法对LeNet5模型进行最大压缩后，参数规模可减少97%，识别准确率降低了不到2个百分点，而压缩后的AlexNet模型参数规模可减少95%，识别准确率提高了6.72个百分点，在保证卷积神经网络精度的前提下，可大幅减少模型的参数量。

关键词: 卷积神经网络, 参数规模, 分组卷积, 卷积拆分, 全局平均池化

Abstract: In order to solve the problem that it is difficult to deploy convolutional neural network model on embedded devices with limited computing and storage resources due to the increasing scale of parameters, a convolutional neural network model compression method is proposed to reduce the scale of parameters. It is found that the number of convolution layer parameters is related to the number of input and output feature maps and the size of convolution kernel, while the number of full connection layer parameters is large and difficult to be reduced significantly. The number of input and output feature maps is reduced by grouping convolution, and the convolution kernel size is reduced by convolution resolution. At the same time, the global average pooling layers are used to replace the fully connected layers to solve the problem of large number of parameters in the fully connected layers. The above methods are applied to LeNet5 and AlexNet for experiments, the experimental results show that the parameters of LeNet5 model can be reduced by 97% and the recognition accuracy can be reduced by less than 2 percentage points by using the combined compression method, the parameters of AlexNet model can be reduced by 95% and the recognition accuracy can be improved by 6.72 percentage points after compression. On the premise of ensuring the accuracy of convolutional neural network, the parameters of the model can be greatly reduced.

Key words: convolutional neural networks, parameter scale, grouping convolution, convolution resolution, global average pooling

朱雪晨, 陈三林, 蔡刚, 黄志洪. 降低参数规模的卷积神经网络模型压缩方法[J]. 计算机与现代化, 2021, 0(09): 83-89.

ZHU Xue-chen, CHEN San-lin, CAI Gang, HUANG Zhi-hong. Compression Method of CNN Model for Parameter Reduction[J]. Computer and Modernization, 2021, 0(09): 83-89.

参考文献

［1］ BALOG M, GAUNT A L, BROCKSCHMIDT M, et al. Deepcoder: Learning to write programs［J］. Machine Learning, 2017， arXiv:1611.01989.
［2］ ERHAN D, SZEGEDY C, TOSHEV A, et al. Scalable object detection using deep neural networks［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014:2155-2162.
［3］ NAKJAI P, KATANYUKUL T. Hand sign recognition for Thai finger spelling: An application of convolution neural network［J］. Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology, 2019,91(2):131-146.
［4］朱亮. 卷积神经网络的全可编程SOC实现［D］. 昆明：云南大学, 2017.
［5］邹蕾,张先锋. 人工智能及其发展应用［J］. 信息网络安全, 2012(2):11-3.
［6］ HAILESELLASIE M, HASAN S R, KHALID F, et al. FPGA-based convolutional neural network architecture with reduced parameter requirements［C］// 2018 IEEE International Symposium on Circuits and Systems. 2018:1-5.
［7］ CHENG Y, WANG D, ZHOU P, et al. Model compression 〖HJ1.08mm〗and acceleration for deep neural networks: The principles, progress, and challenges［J］. IEEE Signal Processing Magazine, 2018,35(1):126-136.
［8］杨扬,蓝章礼,陈巍. 基于统计分析的卷积神经网络模型压缩方法［J］. 计算机系统应用, 2018,27(8):53-59.
［9］邹月娴,余嘉胜,陈泽晗,等. 图像分类卷积神经网络的特征选择模型压缩方法［J］. 控制理论与应用, 2017,34(6):746-752.
［10］黄芬芬. 深度卷积神经网络模型压缩方法研究及应用［D］. 北京:北京邮电大学, 2018.
［11］DENTON E L, ZAREMBA W, BRUNA J, et al. Exploiting linear structure within convolutional networks for efficient evaluation［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014:1269-1277.
［12］JIAO L, LUO C, CAO W, et al. Accelerating low bit-width convolutional neural networks with embedded FPGA［C］// 2017 27th International Conference on Field Programmable Logic and Applications. 2017:1-4.
［13］SAHA S, VARMA G, JAWAHAR C V. Compressing deep neural networks for recognizing places［C］// 2017 4th IAPR Asian Conference on Pattern Recognition. 2017:352-357.
［14］HAN S, MAO H, DALLY W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding［J］. Computer Vision and Pattern Recognition, 2016， arXiv:1510.00149.
［15］WEI X, CHEN H, LIU W C, et al. Mixed-precision quantization for CNN-based remote sensing scene classification［J］. IEEE Geoscience and Remote Sensing Letters, 2020(99):1-5.
［16］QIU J T, WANG J, SONG Y, et al. Going deeper with embedded FPGA platform for convolutional neural network［C］// Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 2016:26-35.
［17］ARKAH Z M, ALZUBAIDI L S. Convolutional neural network with global average pooling for image classification［C］// International Conference on Electrical, Communication, Electronics, Instrumentation and Computing. 2020:171-180.
［18］PANG Y W, SUN M L, JIANG X C, et al. Convolution in convolution for network in network［J］. IEEE Transactions on Neural Networks and Learning Systems, 2016(99):1587-1597.
［19］SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015:1-9.
［20］HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:770-778.
［21］KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks［J］. Communications of the ACM, 2017,60(6):84-90.
［22］YANG Z J, LEI WANG L, LUO L, et al. Bactran: A hardware batch normalization implementation for CNN training engine［J］. IEEE Embedded Systems Letters, 2021,13(1):29-32.

[1]	何思达, 陈平华. 基于意图的轻量级自注意力序列推荐模型[J]. 计算机与现代化, 2024, 0(12): 1-9.
[2]	张晓东1, 白广芝1, 李敏1, 李昊洋2. 基于经验小波变换的油气井产量预测模型 [J]. 计算机与现代化, 2024, 0(12): 53-58.
[3]	刘宝宝, 杨菁菁, 陶露, 王贺应. 基于注意力的DSMSC的遥感图像场景分类[J]. 计算机与现代化, 2024, 0(12): 72-77.
[4]	陈雪松1, 李衡1, 王浩畅2. 结合注意力机制和Mengzi模型的短文本分类[J]. 计算机与现代化, 2024, 0(09): 101-106.
[5]	高帅鹏, 王怡凡. 基于图像的群体情绪识别综述[J]. 计算机与现代化, 2024, 0(08): 98-107.
[6]	周宪溪, 牟莉. 基于改进TF-IDF和AGLCNN的新闻长文本分类模型[J]. 计算机与现代化, 2024, 0(08): 120-126.
[7]	杨江1, 孙晓梅1, 许韬2. 基于业务内容构建股票关联关系的股价预测[J]. 计算机与现代化, 2024, 0(07): 21-25.
[8]	刘存莉1, 雷占占2, 郑澳2. 基于循环卷积神经网络的排水管网缺陷检测方法[J]. 计算机与现代化, 2024, 0(07): 26-35.
[9]	李珊, 王林娜, 高丁佳, 宣海波. 基于图神经网络的多层银企网络融合研究[J]. 计算机与现代化, 2024, 0(05): 27-32.
[10]	钟海龙1, 2, 何月顺1, 何璘琳1, 陈杰1, 田鸣3, 郑瑞银4. 基于代价敏感卷积神经网络的加密流量分类#br# #br#[J]. 计算机与现代化, 2024, 0(05): 55-60.
[11]	高埂1, 肖风丽2, 杨飞1. 基于改进MobileNetV3-Small的色素减退性皮肤病诊断[J]. 计算机与现代化, 2024, 0(05): 120-126.
[12]	游嘉靖1, 2, 何月顺1, 何璘琳1, 钟海龙1, 2. 基于AHP-CNN的加密流量分类方法[J]. 计算机与现代化, 2024, 0(04): 83-87.
[13]	许跃雯1, 李明1, 李莉2. 基于对比学习MocoV2的COVID-19图像分类#br#[J]. 计算机与现代化, 2024, 0(02): 81-87.
[14]	周成诚, 曾庆军, 杨康, 胡家铭, 韩春伟. 基于高效通道注意力模块的运动想象脑电识别[J]. 计算机与现代化, 2023, 0(12): 19-23.
[15]	刘付琪, 张达, 宋建华, 王海东. 基于CNN-BiLSTM的液压系统故障诊断[J]. 计算机与现代化, 2023, 0(09): 10-19.