Traffic-Oriented Shellcode Detection Based on VSM

Abstract

Shellcode is the core part of an attacker exploiting a vulnerability in a binary program, and it is an essential piece of binary bytes to gain control of the target machine. Therefore, the detection of Shellcode is an important part of binary program security protection. However, the currently common static analysis and simulation execution methods for Shellcode detection have problems of low accuracy and low efficiency, resulting in limited actual role. Machine learning models have strong learning and generalization capabilities, and can extract hidden features that are difficult to find manually. This paper proposes a system for detecting Shellcode in network traffic based on the VSM machine learning model. Through the VSM model, the payload data in the network traffic is matched with the Shellcode library to achieve the effect of detecting unknown Shellcode. The experimental results show that the Shellcode detection system based on the VSM model proposed in this paper can effectively detect the known Shellcode, and still has a certain ability to detect the unknown Shellcode.