Systems and methods for importing data from electronic data files

Patent metadata

Field	Value
Publication number	US-9946776-B1
Application number	US-201615292075-A
Country	US
Kind code	B1
Filing date	Oct 12, 2016
Priority date	Sep 4, 2015
Publication date	Apr 17, 2018
Grant date	Apr 17, 2018

Abstract

Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, source electronic data files are received at a data importation system and managed by the data importation system. The data importation system may load detector/transformer plugins and determined whether any of the load detector/transformer plugins apply to the received source electronic data files. The data importation system may also generate transformed data using a transformer corresponding to a detector that applies to the source electronic data files. The data importation system may import the transformed data into at least one of a data analysis system and a database.

First claim

What is claimed is:

1. A system comprising:
one or more memory devices that store a set of instructions; and
at least one processor that executes the set of instructions to perform operations for generating an updated transformed electronic data file using a detector/transformer plugin, the detector/transformer plugin including a detector and a transformer, the detector configured to detect whether an updated electronic source data file is compatible with the transformer, the transformer configured to transform the updated source electronic data file into the updated transformed electronic data file, the operations comprising:
receiving the source electronic data file;
receiving an update to the source electronic data file as an updated electronic source data file;
loading the detector/transformer plugin from a database based on a detection by the detector of the detector/transformer plugin that the updated electronic source data file is compatible with the transformer, the detection based on a file type of the updated electronic source data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the transformer of the detector/transformer plugin; and
providing the updated transformed electronic data file for importing into a data analysis system.

2. The system of claim 1 , wherein an order in which detectors of a plurality of detector/transformer plugins in the database are determined to apply to the updated source electronic data file is determined based on priorities assigned to the plurality of detector/transformer plugins.

3. The system of claim 1 , the operations further comprising storing transformed data in the updated transformed electronic data file.

4. The system of claim 1 , the operations further comprising:
receiving a request to delete the source electronic data file; and
deleting the source electronic data file.

5. The system of claim 4 , the operations further comprising providing a request to the data analysis system to delete transformed data corresponding to the source electronic data file.

6. A method comprising:
performing operations for generating an updated transformed electronic data file using a detector/transformer plugin, the detector/transformer plugin including a detector and a transformer, the detector configured to detect whether an updated electronic source data file is compatible with the transformer, the transformer configured to transform the updated source electronic data file into the updated transformed electronic data file, the operations comprising:
receiving the source electronic data file;
receiving an update to the source electronic data file as an updated electronic source data file;
loading the detector/transformer plugin from a database based on a detection by the detector of the detector/transformer plugin that the updated electronic source data file is compatible with the transformer, the detection based on a file type of the updated electronic source data file; and
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the transformer of the detector/transformer plugin; and
providing the updated transformed electronic data file for importing into data analysis system, one or more modules incorporated into one or more memories of a computer system to configure one or more processors of the computer system to perform the generating of the updated transformed electronic data file using the detector/transformer plugin.

7. The method of claim 6 , wherein an order in which detectors of a plurality of detector/transformer plugins in the database are determined to apply to the updated source electronic data file is determined based on priorities assigned to the plurality of detector/transformer plugins.

8. The method of claim 6 , the operations further comprising storing transformed data in the updated transformed electronic data file.

9. The method of claim 6 , the operations further comprising:
receiving a request to delete the source electronic data file; and
deleting the source electronic data file.

10. The method of claim 9 , the operations further comprising providing a request to the data analysis system to delete transformed data corresponding to the source electronic data file.

11. The method of claim 6 , further comprising providing the transformed updated electronic data file to the data analysis system.

12. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of a machine, cause the machine to perform operations for generating an updated transformed electronic data file using a detector/transformer plugin, the detector/transformer plugin including a detector and a transformer, the detector configured to detect whether an updated electronic source data file is compatible with the transformer, the transformer configured to transform the updated source electronic data file into the updated transformed electronic data file, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as an updated electronic source data file;
loading the detector/transformer plugin from a database based on a detection by the detector of the detector/transformer plugin that the updated electronic source data file is compatible with the transformer, the detection based on a file type of the updated electronic source data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the transformer of the detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.

Assignees

• Palantir Technologies Inc

Inventors

• Yazicioglu Stephen
• Luck Christopher
• Giardina Robert
• Streufert Justin
• Slatcher Timothy
• O'Connor Gregory
• Marc-Aurele Brandon
• Zhu Olivia
• Schindel Howard
• Tung Henry
• Ray Lucas
• Leech Christopher
• Jeney Eric
• Negrus Stefan
• Lee Jason
• Mingione Alessandro
• Doyle John McKinstry
• Pitelka Hunter
• Lozano Ethan
• Ossher Joel
• Fedderly Matthew

Classifications

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F40/186

  Templates · CPC title

• G06F16/245

  Query processing · CPC title

• G06F16/235

  Update request formulation · CPC title