Abstract: There are two major issues in the current Trojan detection system: unable to detect unknown Trojans and low detection rate. To solve these problems, a Trojan horse detection model based on nonlinear SVM by using an effective feature selection optimization algorithm is presented. This approach extracts the API calls sequence of each executable program as feature vector, and by choosing the parts of high degree of differentiation in the feature selection optimization algorithm, quantizes it into identifiable data, and builds SVM feature vector library. SVM classifier is trained with the training dataset to find the optimal classification hyperplane. Experiment results demonstrate that this method is effective and steady in detection capability.

Key words: Trojan detection, SVM, SVM feature vector library, nonlinear SVM classifier