System and method for file type identification using machine learning

Patent metadata

Field	Value
Publication number	US-12436920-B2
Application number	US-202318449617-A
Country	US
Kind code	B2
Filing date	Aug 14, 2023
Priority date	Oct 30, 2017
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

A system and method for file type identification involving extraction of a file-print of a file, the file-print being a unique or practically-unique representation of statistical characteristics associated with the distribution of bits in the binary contents of the file, similar to a fingerprint. The file-print is then passed to a machine learning algorithm that has been trained to recognize file types from their file-prints. The machine learning algorithm returns a predicted file type and, in some cases, a probability of correctness of the prediction. The file may then be encoded using an encoding algorithm chosen based on the predicted file type.

First claim

What is claimed is:

1. A system for identifying a file type comprising:
a computing device comprising a processor, a memory, and a non-volatile data storage device;
a file-print extractor comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the computing device to:
segment an entire file into groups of bytes;
generate a statistical file-print for file type identification of the entire file, the statistical file-print comprising a plurality of statistical characteristics of a distribution of the groups of bytes across the entire file, wherein the statistical file-print comprises at least a mean value and a variance value for the distribution of bytes in the file; and

a file classifier comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to;
process the statistical file-print through a trained machine learning classifier to identify a file type of the file;
wherein the trained machine learning classifier is specifically trained using a plurality of training datasets comprising statistical file-prints derived from files of known types; and
wherein the trained machine learning classifier determines a file type based on statistical patterns in the file-print that correspond to patterns previously identified in files of known type during training.

2. The system of claim 1 , further comprising a codebook database stored on the non-volatile data storage device, the codebook database comprising a plurality of codebooks which may be used to encode or decode files, wherein the identification of the file type is used to select a codebook for encoding or decoding the file.

3. The system of claim 2 , wherein one or more encoding or decoding parameters are configured based on the identification of the file type.

4. The system of claim 2 , further comprising a file signature database stored on the non-volatile data storage device, the file signature database comprising known file signatures for a plurality of file types, wherein:
the file is checked for a file signature prior to file-print extraction;
if a file signature is found, the file signature is compared to known file signatures in the file signature database; and
if the file signature matches a known file signature, a codebook is selected based on the file signature, and the file-print extractor and file classifier are instructed to cease operations on the file.

5. A method for identifying a file type comprising the steps of:
segmenting an entire file into groups of bytes;
generating a statistical file-print for file type identification of the entire file, the statistical file-print comprising a plurality of statistical characteristics of a distribution of the groups of bytes across the entire file, wherein the statistical file-print comprises at least a mean value and a variance value for the distribution of bytes in the file; and
processing the statistical file-print through a trained machine learning classifier specifically trained to recognize file types based on statistical characteristics of entire files to identify a file type of the file.

6. The method of claim 5 , further comprising the steps of using the identification of the file type to select a codebook for encoding or decoding the file from a codebook database stored on the non-volatile data storage device, the codebook database comprising a plurality of codebooks which may be used to encode or decode files.

7. The method of claim 6 , further comprising the step of configuring one or more encoding or decoding parameters based on the identification of the file type.

8. The method of claim 6 , further comprising the steps of:
checking the file for a file signature prior to file print extraction;
if a file signature is found, comparing the file signature to known file signatures in a file signature database stored on the non-volatile data storage device, the file signature database comprising known file signatures for a plurality of file types; and
if the file signature matches a known file signature, selecting a codebook based on the file signature, and instructing the file print extractor and file classifier to cease operations on the file.

Assignees

• Atombeam Technologies Inc

Inventors

• Cooper Joshua
• Riahi Aliasghar
• Haddad Mojgan
• Riahi Ryan Kourosh
• Riahi Razmin
• Yeomans Charles

Classifications

• G06F3/0608

  Saving storage space on storage systems · CPC title

• G06F3/0641

  De-duplication techniques · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• H04L63/1458

  Denial of Service · CPC title

• H03M7/6088

  according to the data type · CPC title