Abstract: In the face of increasingly complex network traffic and data structures with increasing dimensions， the existing traffic anomaly detection schemes have problems such as high false positive rate， low efficiency and poor practicability. To solve these problems， an autoencoder based network traffic anomaly detection model is proposed. Firstly， the model extracts the features of network traffic based on random forest algorithm and selects the optimal collection， and divides the feature vector set into several subsets by hierarchical clustering algorithm to reduce the feature dimension. Then the feature subset is processed in parallel by the autoencoder and the RMSE value is calculated. The maximum average RMSE value of multiple experiments is defined as the normal flow threshold. The average RMSE value and threshold of the test data are used to determine the abnormal traffic. The experimental results show that the recall rate of this model is 4.3 percentage points higher than that of the traditional anomaly detection method， and the running time is reduced by about 37%.

Key words: anomaly detection, autoencoder, hierarchical clustering, random forest algorithm