深度自动编码器的研究与展望

doi:10.3969/j.issn.1006-2475.2014.08.028

摘要/Abstract

摘要：

深度学习是机器学习的一个分支，开创了神经网络发展的新纪元。作为深度学习结构的主要组成部分之一，深度自动编码器主要

用于完成转换学习任务，同时在无监督学习及非线性特征提取过程中也扮演着至关重要的角色。首先介绍深度自动编码器的发展由来、

基本概念及原理，然后介绍它的构建方法以及预训练和精雕的一般步骤，并对不同类型深度自动编码器进行总结，最后在深入分析深度

自动编码器目前存在的问题的基础上，对其未来发展趋势进行展望。

关键词: 深度学习, 深度自动编码器, 预训练, 精雕, 神经网络

Abstract:

Deep learning, which is a branch of machine learning, inaugurates new era in the development of neural

network. As a key component of deep structure, the deep auto-encoder is used to fulfill a task of transforming

learning and plays important role in both unsupervised learning and non-linear characters extraction. We firstly

introduced the origin of deep auto-encoder as well as its basic concept and principle, secondly, the construction

procedure, pre-training and fine-tune procedure of depth auto-encoders were generally introduced, meanwhile, a

comprehensive summarization of different kinds of DAE was made. At last, the direction of future work was proposed

based on an in-depth study of current DAE researches.

Key words: deep learning, deep auto-encoder(DAE), pre-train, fine-tune, neural network

中图分类号:

TP391.9

曲建岭，杜辰飞，邸亚洲，高峰，郭超然. 深度自动编码器的研究与展望[J]. 计算机与现代化, doi: 10.3969/j.issn.1006-2475.2014.08.028.

QU Jian-ling， DU Chen-fei， DI Ya-zhou， GAO Feng， GUO Chao-ran. Research and Prospect of Deep Auto-encoders[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2014.08.028.

参考文献［43］

［1］	Arel L, Rose D C, Karnowski T P. Deep machine learning a new frontier in artificial intelligence research［J］.
	Computational Intelligence Magazine, 2010,5(4):13-18.
［2］	Hinton G E, Osinder S, Teh Y W. A fast learning algorithm for deep belief nets［J］. Neural Computation,
20	06,18(7):1527-1554.
［3］	Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors［J］. Nature,
19	86,323:533-536.
［4］	Bengio Y, Lamblin P, Popovici D, et al. Greedy layer-wise training of deep networks［C］. Proc. of the 20th
	Annual Conference on Neural Information Processing System. 2006:153-160.
［5］	Vincent p, Larochelle H, Bengio Y. Extracting and composing robust features with denoising auto-encoders［C
］	Proc. of the 25th International Conference on Machine Learning. 2008:1096-1103.
［6］	Bengio Y. Learning deep architectures for AI［J］. Foundations and Trends in Machine Learning, 2009,2(1):1
	-127.
［7］	Salah R, Vincent P, Muller X, et al. Contractive auto-encoders: Explicit invariance during feature
	extraction［C］. Proc. of the 28th International Conference on Machine Learning. 2011:833-840.
［8］	Masci J, Meier U, Ciresan D. Stacked convolutional auto-encoders for hierarchical feature extraction［C］.
	Proc. of the 21th International Conference on Artifical Neural Networks. 2011,6791:52-59.
［9］	Guyon I, Dror G, Lemaire V, et al. Auto-encoders unsupervised learning and deep architectures［C］. Proc.
	of the 28th International Conference on Machine Learning, 2012:37-50.
［10］	Vincent P, Larochelle H, Lajoie I, et al. Stacked denoising auto-encoders: Learning useful representations
	in a deep network with a local denoising criterion［J］. Joural of Machine Learning Research, 2010,11:3371-3408.
［11］	Mitchell B, Sheppard J. Deep structure learning: Beyond connectionist approaches［C］. Proc. of the 11th
	International Conference on Machine Learning and Applications. 2012:162-167.
［12］	Erhan D, Bengio Y, Couville A, et al. Why does unsupervised pre-training help deep learning［J］. Journal
	of Machine Learning Research, 2010,11:625-660.
［13］	Li Deng, Selzer M L, Yu Dong, et al. Binary coding of speech spectrograms using a deep auto-encoder［C］.
	Proc. of the 11th International Speech Communication Association. 2010:1692-1695.
［14］	Lee H, kanadhamk C E. A sparse deep belief net model for visual area［C］. Proc. of the 21th Neural
	Information Process Systems International Conference. 2007:873-880.
［15］	Amaral T, Silva L M, Alexande L A, et al. Using different cost functions to train stacked auto- encoders［
C］	Proc. of the 12th Mexican International Conference on Artificial Intelligence. 2013:114-120.
［16］	Bengio Y, Delalleau O. On the expressive power of deep architectures［C］. Proc. of the 22nd International
	Conference on Algorithmic Learning Theory. 2011:18-36.
［17］	Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks［J］. Science,
20	06,313:504-507.
［18］	Gehring J, Miao Yajie, Metze F, et al. Extracting deep bottleneck features using stacked auto-encoders［C
］	Proc. of the 26th IEEE International Conference on Acoustics, Speech and Signal Processing. 2013:3377-3381.
［19］	Lange S, Riedmiller M. Deep auto-encoder neural networks in reinforcement learning lange［C］. Proc. of
	the International Joint Conference on Neural Networks. 2010:18-23.
［20］	Suk H, Lee S W, Shen Dinggang. Latent feature representation with stacked auto-encoder for AD/MCI
	diagnosis［J］. Brain Structure Function, 2013,218(6):1017-1036.
［21］	Iamsat S, Horata P. Handwritten Character Recognition Using Histograms of Oriented Gradient Features in
	Deep Learning of Artificial Neural Network［C］. Proc. of the 3rd International Conference on IT Convergence and
	Security. 2013:1-5.
［22］	Gupta M, Lam S M. Weight decay back propagation for noisy data［J］. Neural Network, 1998,11(6):1127-1137.
	
［23］	李海峰,李纯果. 深度学习结构和算法比较分析［J］. 河北大学学报(自然科学版), 2012,32(5):538-543.
［24］	Luo Xuxi, Li Wan. A novel efficient method for training sparse auto-encoders［C］. Proc. of the 6th
	International Congress on Image and Signal Processing. 2013:1019-1023.
［25］	Deng Jun, Zhang Zixing, Erik M, et al. Sparse auto-encoder based feature transfer learning for speech
	emotion recognition［C］. Proc. of Humaine Association Conference on Affective Computing and Intelligent
	Interaction. 2013:511-516.
［26］	Ma Yunlong, Zhang Peng, Gao Yanan. Parallel auto-encoder for efficient outlier detection［C］// Proceeding
	of IEEE International Conference on Big Data. 2013:15-17.
［27］	张开旭,周昌乐. 基于自动编码器的中文词汇特征无监督学［J］. 中文信息学报, 2013,27(5):1-7,92.
［28］	Y Yang, S Guang, Shah M. Semi-supervised learning of feature hierarchies for object detection in a video［
C］	// Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition. 2013:1650-1657.
［29］	Wong W K, S Mingming. Deep learning regularized fisher mapping［J］. Neural Networks, 2011,22(10):1668-
16	75.
［30］	C Minmin, Killan Q W, Xu Zhixiang. Marginalized denoising auto-encoders for domain adaptation［C］. Proc.
	of the 29th International Conference on Machine Learning. 2012:538-542.
［31］	Q Hao, Y Zhe, Z Yajin. A new training principle for stacked denoising auto-encoders［C］. Proc. of the 7th
	International Conference on Image and Graphics. 2013:384-389.
［32］	Schulz H, Cho K, Raiko T, et al. Two layers contractive encoding with shortcuts for semi-supervised
	learning［C］. Proc. of the 20th International Conference on Neural Information Processing. 2013:450-457.
［33］	Gutmann M, Hyvarinen A. Noise-contrastive estimation: a new estimation principle for un-normalized
	statistical models［C］. Proc. of the 13th International Conference on Artificial Intelligence and Statistics.
20	10:297-304.
［34］	Firat O, Vural F T Y. Representation learning with convolutional sparse auto-encoders for remote sensing［
C］	Proc. of the 21st Signal Processing and Communications Applications Conference. 2013:24-26.
［35］	孙志军,薛磊,许阳明. 深度学习研究综述［J］. 计算机应用研究, 2012,29(8):2806-2810.
［36］	庄永文. 关于系数编码理论及其应用［J］. 现代电子技术, 2008(7):157-160.
［37］	刘海宁. 基于稀疏编码的设备状态识别及其重型轧辊磨床监测应用［D］. 上海:上海交通大学, 2011.
［38］	孙志军,薛磊,许阳明. 基于深度学习的边际Fisher分析特征提取算法［J］. 电子与信息报, 2013,35(4):805-811.
［39］	殷力昂. 一种在深度结构中学习原型的分类方法［D］. 上海:上海交通大学, 2012.
［40］	Hinton G E, Krizhevsky A, Wang S. Transfor-ming auto-encoders［C］. Proc. of Internet Corporation for
	Assigned Names and Numbers. 2011:387-396.
［41］	夏丁胤. 互联网图像高效标注和解译的关键技术研究［D］. 杭州:浙江大学, 2010.
［42］	Hinton G E, Li Deng, Dahl G E. Deep neural networks for acoustic modeling in speech recognition: The
	shared views of four research groups［J］. Signal Processing, Magazine, 2012, 29(6), 82-97.
［43］	Guyon G, Dror V, Lemaire G. Auto-encoders, unsupervised learning, and deep architecture［J］. Journal of
	Machine Learning Research, 2012,13:37-49.

[1]	胡崇佳, 刘金洲, 方立. 基于无监督域适应的室外点云语义分割[J]. 计算机与现代化, 2024, 0(01): 74-79.
[2]	林威. 基于自监督学习和数据回放的新闻推荐模型增量学习方法[J]. 计算机与现代化, 2023, 0(12): 1-6.
[3]	宋涛涛, 李艳萍, 李洪港, 韩春雪. 基于改进变结构趋近律的机械臂滑模控制系统[J]. 计算机与现代化, 2023, 0(12): 14-18.
[4]	周成诚, 曾庆军, 杨康, 胡家铭, 韩春伟. 基于高效通道注意力模块的运动想象脑电识别[J]. 计算机与现代化, 2023, 0(12): 19-23.
[5]	梁天恺, 黄康华, 刘凯航, 兰岚, 曾碧. 基于双向同态加密的深度联邦图片分类方法[J]. 计算机与现代化, 2023, 0(12): 36-40.
[6]	邱凯星, 冯广. 基于双重特征注意力的多标签图像分类模型[J]. 计算机与现代化, 2023, 0(12): 41-47.
[7]	张伯泉, 麦海鹏, 陈嘉敏, 逄锦聚. 基于高灰度值注意力机制的脑白质高信号分割[J]. 计算机与现代化, 2023, 0(12): 67-75.
[8]	王宇航, 董宝良, 公超, 尚真真, 姚康宁. 基于意图识别的空中群目标动态威胁评估[J]. 计算机与现代化, 2023, 0(12): 100-104.
[9]	马泽宇, 叶宁, 徐康, 王甦, 王汝传, . 基于FMCW雷达和ResNeSt-GRU的行为识别方法[J]. 计算机与现代化, 2023, 0(11): 101-107.
[10]	谷明轩, 范冰冰. 基于多模态特征融合的抑郁症识别[J]. 计算机与现代化, 2023, 0(10): 17-22.
[11]	陈俊义. 基于图节点动静态特征的健康事件预测模型[J]. 计算机与现代化, 2023, 0(10): 39-44.
[12]	李延满, 王必恒, 赵羚焱. 基于轻量化YOLOv5的安全帽检测[J]. 计算机与现代化, 2023, 0(10): 59-64.
[13]	黎世达, 项剑文. 一种提高图像识别模型鲁棒性的弱化强化方法[J]. 计算机与现代化, 2023, 0(10): 70-76.
[14]	陈子健, 段春红. 面向在线学习情境的认知情绪面部表情识别[J]. 计算机与现代化, 2023, 0(10): 92-98.
[15]	沈加炜, 陆一鸣, 陈晓艺, 钱美玲, 陆卫忠, . 基于深度学习的人体行为检测方法研究综述[J]. 计算机与现代化, 2023, 0(09): 1-9.