Static feature extraction from structured files

Patent metadata

Field	Value
Publication number	US-10838844-B2
Application number	US-201916424261-A
Country	US
Kind code	B2
Filing date	May 28, 2019
Priority date	Jan 31, 2014
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

Data is received or accessed that includes a structured file encapsulating data required by an execution environment to manage executable code wrapped within the structured file. Thereafter, code and data regions are iteratively identified in the structured file. Such identification is analyzed so that at least one feature can be extracted from the structured file. Related apparatus, systems, techniques and articles are also described.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving or accessing data comprising a structured file encapsulating data required by an execution environment to manage executable code wrapped within the structured file;
iteratively identifying code and data regions in the structured file for parsing and disassembly; and
extracting a plurality of features from the structured file based on the parsing and disassembly of at least a portion of the identified code and data regions;
truncating one of the features;
generating an additional feature noting a reason for the truncation, the reason being one of the data comprising non-printable characters, invalid encodings, or a mixture of languages;
inputting the extracted features and the additional feature into at least one machine learning model to classify the structured file as malicious or safe; and
preventing access or execution of the structured file if it is classified as malicious; or accessing or executing the structured file if it is classified as safe.

2. The method of claim 1 , wherein the structured file is selected from a group consisting of: Portable Executable (PE) format files, disk operating system (DOS) executable files, New Executable (NE) files, Linear Executable (LE) files, Executable and Linkable Format (ELF) files, JAVA Archive (JAR) files, and SHOCKWAVE/FLASH (SWF) files.

3. The method of claim 1 , wherein the execution environment is an operating system or a virtual machine.

4. The method of claim 1 further comprising:
determining that the structured file is valid by examining the first header or the second header within the structured file to determine whether it encapsulates a valid signature.

5. The method of claim 1 , wherein the extracted features comprise a first order feature.

6. The method of claim 5 further comprising:
deriving the extracted at least one first order feature into a higher-order feature.

7. The method of claim 1 further comprising:
analyzing negative space within the structured file to extract at least one additional feature, the negative space being different from the identified code and data regions.

8. The method of claim 1 further comprising:
transforming at least one of the extracted features.

9. The method of claim 8 , wherein the transforming comprises:
sanitizing at least one of the extracted features by encoding associated data in an unambiguously reversible way.

10. The method of claim 9 , wherein the transforming comprises:
truncating at least one of the extracted features when a size of such at least one feature exceeds a predetermined amount.

11. The method of claim 1 , wherein the code and data regions in the structured file are identified by:
discovering a structure of the structured file;
analyzing and checking a first header located at a root of the structure;
analyzing and checking a second header in the structure referenced by the first header in response to the checking of the first header; and
looping through remaining portions of the structure other than the first header and the second header until all code start points for analysis and disassembly are identified.

12. The method of claim 1 , wherein data within the structured file is arranged hierarchically and the structured file comprises a top level header encapsulating a first data structure that encapsulates a second data structure.

13. The method of claim 1 , wherein the first header is a disk operating system (DOS) header and the second header is a Portable Executable (PE) header.

14. A computer-implemented method for extraction of features from a structured file, the method comprising:
receiving or accessing data comprising a structured file encapsulating data required by an execution environment to manage executable code wrapped within the structured file;
determining that the structured file is valid;
iteratively analyzing code and data within the structured file to identify and extract first order features;
analyzing negative space within the structured file to identify and extract additional first order features;
transforming a portion of the extracted first order features into higher order features using one or more transformation techniques;
truncating at least one of the first order features;
generating an additional feature characterizing the truncating;
inputting the transformed higher order features and non-transformed first order features and the generated additional feature into a machine learning model trained to classify the structured file as malicious or benign;
providing, by the machine learning model, output characterizing the classification of the structured file; and
preventing access or execution of the structured file if it is classified as malicious; or accessing or executing the structured file if it is classified as safe.

15. The method of claim 14 , wherein the iteratively analyzing code and data within the structured file to identify and extract first order features comprises:
discovering a structure of the structured file;
analyzing and checking a first header located at a root of the structure;
analyzing and checking a second header in the structure referenced by the first header in response to the checking of the first header; and
looping through remaining portions of the structure other than the first header and the second header until all code start points for analysis are identified.

16. A system comprising:
at least one data processor; and
memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
receiving or accessing data comprising a structured file encapsulating data required by an execution environment to manage executable code wrapped within the structured file;
iteratively identifying code and data regions in the structured file for parsing and disassembly; and
extracting a plurality of features from the structured file based on the parsing and disassembly of at least a portion of the identified code and data regions;

truncating one of the features;
generating an additional feature noting a reason for the truncation, the reason being one of the data comprising non-printable characters, invalid encodings, or a mixture of languages;
inputting the extracted features and the additional feature into at least one machine learning model to classify the structured file as malicious or safe; and
preventing access or execution of the structured file if it is classified as malicious; or accessing or executing the structured file if it is classified as safe.

17. The system of claim 16 , wherein the structured file is selected from a group consisting of: Portable Executable (PE) format files, disk operating system (DOS) executable files, New Executable (NE) files, Linear Executable (LE) files, Executable and Linkable Format (ELF) files, JAVA Archive (JAR) files, and SHOCKWAVE/FLASH (SWF) files.

18. The system of claim 17 , wherein the execution environment is a virtual machine.

19. The system of claim 16 , wherein the operations further comprise:
transforming at least one of the extracted features comprising sanitizing at least one of the extracted features by encoding associated data in an unambiguously reversible way.

20. The system of claim 16 , wherein the code and data regions in the structured file are identified by:
discovering a structure of the structured file;
analyzing and checking a first header located at a root of the structure;
analyzing and checking a

Assignees

• Cylance Inc

Inventors

• Soeder Derek A
• Permeh Ryan
• Golomb Gary
• Wolff Matthew

Classifications

• G06F21/562 Primary

  Static detection · CPC title

• G06F40/205

  Parsing · CPC title

• G06F16/188

  Virtual file systems · CPC title

• G06F11/36

  Prevention of errors by analysis, debugging or testing of software · CPC title

• G06F16/116

  Details of conversion of file system types or formats · CPC title