计算机与现代化 ›› 2022, Vol. 0 ›› Issue (09): 51-59.
出版日期:
2022-09-22
发布日期:
2022-09-22
作者简介:
李屹(1984—),男,山东滨州人,博士后,博士,研究方向:图像处理,计算机视觉,E-mail: yili6251@163.com; 魏建国(1971—),男,教授,博士生导师,博士,研究方向:智能人机交互,声纹识别,语音识别,E-mail: jianguo@tju.edu.cn; 刘贯伟(1983—),男,高级工程师,硕士,研究方向:现金机具设备相关技术模型,图像处理,E-mail: liugw@cashwaytech.com。
Online:
2022-09-22
Published:
2022-09-22
摘要: 模型剪枝算法利用不同的标准或方式对深度神经网络中冗余神经元进行裁剪,在不损失模型精度的情况下对模型进行最大程度的压缩,从而可以减少存储并提升速度。首先,对模型剪枝算法的研究现状与主要研究方向进行总结并归类。主要研究方向包括剪枝的尺度、剪枝元素重要性评估的方法、剪枝的稀疏度、剪枝的理论基础及对于不同任务的剪枝等方面。然后对近年来具有代表性的剪枝算法进行详细描述。最后对此领域的研究提出未来展望。
李屹, 魏建国, 刘贯伟. 模型剪枝算法综述[J]. 计算机与现代化, 2022, 0(09): 51-59.
LI Yi, WEI Jian-guo, LIU Guan-wei. Survey of Model Pruning Algorithms[J]. Computer and Modernization, 2022, 0(09): 51-59.
[1] | HAN S, MAO H Z, DALLY W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding[C]// Proceedings of International Conference on Learning Representation. 2016. |
[2] | DENIL M, SHAKIBI B, DINH L, et al. Prediction parameters in deep learning[C]// Proceedings of Advances in Neural Information Processing Systems. 2013:2148-2156. |
[3] | CUN Y L, DENKER J S, SOLLA S A. Optimal brain damage[C]// Proceedings of Advances in Neural Information Processing Systems. 1990:589-605. |
[4] | HASSIBI B, STORK D G. Second order derivatives for network pruning: Optimal brain surgeon[C]// Proceedings of Advances in Neural Information Processing Systems. 1992:164-171. |
[5] | HAN S, POOL J, TRAN J, et al. Learning both weights and connections for efficient neural networks[C]// Proceedings of Advances in Neural Information Processing Systems. 2015:1135-1143. |
[6] | LIU Z, SUN M J, ZHOU T H, et al. Rethinking the value of network pruning[C]// Proceedings of International Conference on Learning Representation. 2019. |
[7] | MITTAL D, BHARDWAJ S, KHAPRA M M, et al. Recovering from random pruning: On the plasticity of deep convolutional neural networks[C]// Proceedings of IEEE Winter Conference on Applications of Computer Vision. 2018:848-857. |
[8] | WANG Y L, ZHANG X L, XIE L X, et al. Pruning from scratch[C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2020:12273-12280. |
[9] | MANESSI F, ROZZA A, BIANCO S, et al. Automated pruning for deep neural network compress[C]// Proceedings of International Conference on Pattern Recognition. 2018: 654-657. |
[10] | CARREIRA-PERPINAN M A, IDELBAYEV Y. Learning-compression algorithm for neural network pruning[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018:8532-8541. |
[11] | LIN M B, JI R R, ZHANG Y X, et al. Channel pruning via automatic structure search[C]// Proceedings of International Joint Conference on Artificial Intelligence. 2020:673-679. |
[12] | LIU Z H, MU H Y, ZHANG X Y, et al. Meta learning for automatic neural network channel pruning[C]// Proceedings of IEEE International Conference on Computer Vision. 2019:3295-3304. |
[13] | HE Y H, LIN J, LIU Z J, et al. AMC: AutoML or model compression and acceleration on mobile devices[C]// Proceedings of European Conference on Computer Vision. 2018:784-800. |
[14] | LIN S H, JI R R, YAN C Q, et al. Towards optimal structured CNN pruning via generative adversarial learning[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019:2790-2799. |
[15] | NING X F, ZHAO T C, LI W S, et al. DSA: More efficient budgeted pruning via differentiable sparsity allocation[C]// Proceedings of European Conference on Computer Vision. 2020:592-607. |
[16] | GUO S P, WANG Y J, LI Q Q, et al. DMCP: Differentiable Markov channel pruning for neural networks[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:1536-1544. |
[17] | LI Y W, GU S H, MAYER C, et al. Group sparsity: The hinge between filter pruning and decomposition for network compression[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:8018-8027. |
[18] | WANG T Z, WANG K, CAI H, et al. APQ: Joint search for network architecture, pruning and quantization policy[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:2078-2087. |
[19] | TUNG F, MORI G. CLIP-Q: Deep network compression learning by in-parallel pruning-quantization[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018:7873-7882. |
[20] | DELLINGER F, BOULAY T, BARRENECHEA D M, et al. Multi-task network pruning and embedded optimization for real-time deployment in ADAS[J]. arXiv preprint arXiv:2102.07831, 2021. |
[21] | HE W, WU M Q, LIANG M F, et al. CAP: Context-aware pruning for semantic segmentation[C]// Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 2021:960-969. |
[22] | MENG F X, CHENG H, LI K, et al. Pruning filter in filter[C]// Proceedings of Advances in Neural Information Processing Systems. 2020:17629-17640. |
[23] | MAO H Z, HAN S, POOL J, et al. Exploring the regularity of sparse structure in convolutional neural network[J]. arXiv preprint arXiv:1705.08922, 2017. |
[24] | LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient ConvNets[C]// Proceedings of International Conference on Learning Representation. 2017. |
[25] | HE Y, LIU P, WANG Z W, et al. Filter pruning via geometric median for deep convolutional neural network acceleration[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019:4340-4349. |
[26] | WEN W, WU C P, WANG Y D, et al. Learning structured sparsity in deep neural networks[C]// Proceedings of Advances in Neural Information Processing Systems. 2016:2074-2082. |
[27] | LIU Z, LI J G, SHEN Z Q, et al. Learning efficient convolutional networks through network slimming[C]// Proceedings of IEEE International Conference on Computer Vision. 2017:2736-2744. |
[28] | HU H Y, PENG R, TAI Y W, et al. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures[J]. arXiv preprint arXiv:1607.03250, 2016. |
[29] | LIN M B, JI R R, WANG Y, et al. Filter pruning using high-rank feature map[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:1529-1538. |
[30] | LUO J H, WU J X. An entropy-based pruning method for CNN compression[J]. arXiv preprint arXiv:1706.05791, 2017. |
[31] | MOLCHANOV P, TYREE S, KARRAS T, et al. Pruning convolutional neural networks for resource efficient transfer learning[C]// Proceedings of International Conference on Learning Representation. 2017. |
[32] | XIN D, CHEN S Y, PAN S. Learning to prune deep neural networks via layer-wise optimal brain surgeon[C]// Proceedings of Advances in Neural Information Processing Systems. 2017:4857-4867. |
[33] | YOU Z H, YAN K, YE J M, et al. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks[C]// Proceedings of Advances in Neural Information Processing Systems. 2019:2133-2144. |
[34] | LUO J H, WU J X, LIN W Y. ThiNet: A filter level pruning method for deep neural network compression[C]// Proceedings of IEEE International Conference on Computer Vision. 2017:5058-5066. |
[35] | HE Y H, ZHANG X Y, SUN J. Channel pruning for accelerating very deep neural networks[C]// Proceedings of IEEE International Conference on Computer Vision. 2017:1389-1397. |
[36] | KARABOGA D. An idea based on honey bee swarm for numerical optimization[R]. Technical Report, 2005. |
[37] | LI Y W, GU S H, ZHANG K, et al. DHP: Differentiable meta pruning via hypernetworks[C]// Proceedings of European Conference on Computer Vision. 2020:608-624. |
[38] | HE Y, KANG G L, DONG X Y, et al. Soft filter pruning for accelerating deep convolutional neural networks[C]// Proceedings of International Joint Conference on Artificial Intelligence. 2018:2234-2240. |
[39] | LIN M B, JI R R, LI S J, et al. Filter sketch for network pruning[J]. arXiv preprint arXiv:2001.08514, 2020. |
[40] | LIBERTY E. Simple and deterministic matrix sketching[C]// Proceeding of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2013:581-588. |
[41] | GUO Y W, YAO A B, CHEN Y R. Dynamic network surgery for efficient DNNs[C]// Proceedings of Advances in Neural Information Processing Systems. 2016:1379-1387. |
[42] | ZHU M H, GUPTA S. To prune, or not to prune: Exploring the efficacy of pruning for model compression[C]// Proceedings of International Conference on Learning Representations. 2018. |
[43] | LUO J H, WU J X. An end-to-end trainable filter pruning method for efficient deep model inference[J]. arXiv preprint arXiv:1805.08941, 2020. |
[44] | FRANKE J, CARBIN M. The lottery ticket hypothesis: Finding sparse, trainable neural networks[J]. arXiv preprint arXiv:1803.03635, 2018. |
[45] | ZHOU H, LAN J, LIU R, et al. Deconstruction lottery tickets: Zeros, signs, and the super-mask[C]// Proceedings of Advances in Neural Information Processing Systems. 2019:3597-3607. |
[46] | RAMANUJAN V, WORTSMAN M, KEMBHAVI A, et al. What’s hidden in a randomly weighted neural network[C]// Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:11893-11902. |
[47] | MALACH E, YEHUDAI G, SHALEV-SHWARTZ S, et al. Proving the lottery ticket hypothesis: Pruning is all you need[C]// Proceedings of International Conference on Machine Learning. 2020:6682-6691. |
[48] | QIAN X, KLABJAN D. A probabilistic approach to neural network pruning[J]. arXiv preprint arXiv:2105.10065v1, 2021. |
[49] | WANG Y, LU Y D, BLANKEVOORT T. Differentiable joint pruning and quantization for hardware efficiency[C]// Proceedings of European Conference on Computer Vision. 2020:259-277. |
[50] | DAI B, ZHU C, WIPF D. Compressing neural networks using the variational information bottleneck[C]// Proceedings of International Conference on Machine Learning. 2018:1135-1144. |
[51] | GUO J Y, OUYANG W L, XU D. Multi-dimensional pruning: A unified framework for model compression[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:1508-1517. |
[1] | 胡崇佳, 刘金洲, 方 立. 基于无监督域适应的室外点云语义分割[J]. 计算机与现代化, 2024, 0(01): 74-79. |
[2] | 林 威. 基于自监督学习和数据回放的新闻推荐模型增量学习方法[J]. 计算机与现代化, 2023, 0(12): 1-6. |
[3] | 周成诚, 曾庆军, 杨 康, 胡家铭, 韩春伟. 基于高效通道注意力模块的运动想象脑电识别[J]. 计算机与现代化, 2023, 0(12): 19-23. |
[4] | 梁天恺, 黄康华, 刘凯航, 兰 岚, 曾 碧. 基于双向同态加密的深度联邦图片分类方法[J]. 计算机与现代化, 2023, 0(12): 36-40. |
[5] | 邱凯星, 冯 广. 基于双重特征注意力的多标签图像分类模型[J]. 计算机与现代化, 2023, 0(12): 41-47. |
[6] | 张浩洋, 尹梓名, 乐珺怡, 沈达聪, 束翌俊, 杨自逸, 孔祥勇, 龚 伟. 3D-SPRNet: 一种基于并行解码器和双注意力机制的胆囊癌分割模型[J]. 计算机与现代化, 2023, 0(12): 59-66. |
[7] | 张伯泉, 麦海鹏, 陈嘉敏, 逄锦聚. 基于高灰度值注意力机制的脑白质高信号分割[J]. 计算机与现代化, 2023, 0(12): 67-75. |
[8] | 马泽宇, 叶 宁, 徐 康, 王 甦, 王汝传, . 基于FMCW雷达和ResNeSt-GRU的行为识别方法[J]. 计算机与现代化, 2023, 0(11): 101-107. |
[9] | 李延满, 王必恒, 赵羚焱. 基于轻量化YOLOv5的安全帽检测[J]. 计算机与现代化, 2023, 0(10): 59-64. |
[10] | 黎世达, 项剑文. 一种提高图像识别模型鲁棒性的弱化强化方法[J]. 计算机与现代化, 2023, 0(10): 70-76. |
[11] | 沈加炜, 陆一鸣, 陈晓艺, 钱美玲, 陆卫忠, . 基于深度学习的人体行为检测方法研究综述[J]. 计算机与现代化, 2023, 0(09): 1-9. |
[12] | 刘禅奕, 黄 丹, 薛林雁, 王 涛, 朱 桃, . 改进EfficientNet网络的COVID-19 X光分类[J]. 计算机与现代化, 2023, 0(09): 94-99. |
[13] | 马国祥, 杨凌菲, 严传波, 张志豪, 孙 彬, 王晓荣. 基于深度DenseNet网络的肝包虫病超声影像诊断方法[J]. 计算机与现代化, 2023, 0(09): 100-104. |
[14] | 农皓程, 任德均, 任秋霖, 刘澎笠, 黄德成. 基于改进ConvNeXt的软塑包装表面异常检测算法[J]. 计算机与现代化, 2023, 0(08): 12-17. |
[15] | 欧阳飞, 吴 旭, 向东升. 基于改进YOLOX的垃圾分类检测方法[J]. 计算机与现代化, 2023, 0(08): 68-73. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||