Abstract: With the establishment and improvement of the network security mechanism of the government、 the enterprises and the Institutions, the threshold for attacking the target system from the outside is getting higher and higher. So the insider threats are gradually increasing. The internal threats are different from external threats. The attackers are mainly from internal users, so it makes the attacks more concealed and harder to be detected. The paper first analyzes user behaviors in the public SEA data set, then proposes an insider threat detection based on hybrid N-Gram and XGBoost theory, using the big data and machine learning methods. Three feature extraction methods: bag-of-words, N-Gram, and vocabulary are used for experimental comparison and N value experimental screening. The internal threat detection method based on the hybrid N-Gram model and XGBoost algorithm has a better detection effect than one-dimensional data and two-dimensional data. The effect of combining the different features of the four-dimensional data on the feature subset is better. The specificity reaches 0.23, the sensitivity reaches 27.65, the accuracy reaches 0.94, and the F1 value reaches 0.97. Comparing the 4 evaluation indicators of specificity, sensitivity, accuracy, and F1 value, the feature extraction method based on hybrid N-gram is more effective in detection than traditional bag-of-words and vocabulary feature extraction methods. This detection method not only improves the discrimination of internal threat detection signatures, but also improves the accuracy of feature extraction and calculation performance.

Key words: hybrid N-Gram, XGBoost, internal threats, SEA, evaluation index