Systems and methods for importing data from electronic data files

Patent metadata

Field	Value
Publication number	US-10545985-B2
Application number	US-201815927947-A
Country	US
Kind code	B2
Filing date	Mar 21, 2018
Priority date	Sep 4, 2015
Publication date	Jan 28, 2020
Grant date	Jan 28, 2020

Abstract

Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, source electronic data files are received at a data importation system and managed by the data importation system. The data importation system may load detector/transformer plugins and determined whether any of the load detector/transformer plugins apply to the received source electronic data files. The data importation system may also generate transformed data using a transformer corresponding to a detector that applies to the source electronic data files. The data importation system may import the transformed data into at least one of a data analysis system and a database.

First claim

What is claimed is:

1. A system comprising:
one or more memory devices that store a set of instructions; and
at least one processor that executes the set of instructions to perform operations for transforming an updated source electronic data file into an updated transformed electronic data file that is capable of being imported into a data analysis system, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as the updated source electronic data file;
selecting a detector/transformer plugin from a plurality of detector/transformer plugins stored in a database, the selecting including cycling through at least a portion of the plurality of detector/transformer plugins based on priorities associated with the plurality of detector/transformer plugins and determining which of the at least the portion of the plurality of detector/transformer plugins is capable of transforming the updated source electronic data file into the updated transformed electronic data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the selected detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.

2. The system of claim 1 , wherein the cycling through of the at least the portion of the detector/transformer plugins includes:
loading the at least the portion of the plurality of detector/transformer plugins;
determining whether a first detector/transformer of the at least the portion of the detector/transformer plugins applies; and
based on a determination that the first detector/transformer does not apply, moving onto a second detector/transformer of the at least the portion of the detector/transformer plugins.

3. The system of claim 2 , wherein the at least the portion of the detector/transformer plugins is selected based on a priority associated with each of the at least the portion of detector/transformer plugins exceeding a priority threshold.

4. The system of claim 2 , wherein the determination that the first detector/transformer does not apply is based on a determination that a transformer of the detector/transformer plugin is not capable of performing the transforming of the updated source electronic data file into the updated transformed electronic data file.

5. The system of claim 1 , the operations further comprising, based on a determination that the selected detector/transformer is associated with a data mapper, mapping the source electronic data file and storing the mapped version of the source electronic data file to support mapping of a format of the source electronic data file to a canonical format included in a transformation template.

6. The system of claim 1 , wherein the performing the transforming of the updated source electronic data file is based on a selection of a process option included in a graphical user interface.

7. The system 1 , wherein the source electronic data file is a spreadsheet and the selecting of the detector/transformer plugin is based on file types associated with a tab of a spreadsheet.

8. A method comprising:
performing operations for transforming an updated source electronic data file into an updated transformed electronic data file that is capable of being imported into a data analysis system, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as the updated source electronic data file;
selecting the detector/transformer plugin from a plurality of detector/transformer plugins stored in a database, the selecting including cycling through at least a portion of the plurality of detector/transformer plugins based on priorities associated with the plurality of detector/transformer plugins and determining which of the at least the portion of the plurality of detector/transformer plugins is capable of transforming the updated source electronic data file into the updated transformed electronic data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the selected detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.

9. The method of claim 8 , wherein the cycling through of the at least the portion of the detector/transformer plugins includes:
loading the at least the portion of the plurality of detector/transformer plugins;
determining whether a first detector/transformer of the at least the portion of the detector/transformer plugins applies; and
based on a determination that the first detector/transformer does not apply, moving onto a second detector/transformer of the at least the portion of the detector/transformer plugins.

10. The method of claim 9 , wherein the at least the portion of the detector/transformer plugins is selected based on a priority associated with each of the at least the portion of detector/transformer plugins exceeding a priority threshold.

11. The method of claim 9 , wherein the determination that the first detector/transformer does not apply is based on a determination that a transformer of the detector/transformer plugin is not capable of performing the transforming of the updated source electronic data file into the updated transformed electronic data file.

12. The method of claim 8 , the operations further comprising, based on a determination that the selected detector/transformer is associated with a data mapper, mapping the source electronic data file and storing the mapped version of the source electronic data file to support mapping of a format of the source electronic data file to a canonical format included in a transformation template.

13. The method of claim 8 , wherein the performing the transforming of the updated source electronic data file is based on a selection of a process option included in a graphical user interface.

14. The method of claim 8 , wherein the source electronic data file is a spreadsheet and the selecting of the detector/transformer plugin is based on file types associated with a tab of a spreadsheet.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of a machine, cause the machine to perform operations for transforming an updated source electronic data file into an updated transformed electronic data file that is capable of being imported into a data analysis system, the operations comprising:
receiving a source electronic data file;
receiving an update to the source electronic data file as the updated source electronic data file;
selecting the detector/transformer plugin from a plurality of detector/transformer plugins stored in a database, the selecting including cycling through at least a portion of the plurality of detector/transformer plugins based on priorities associated with the plurality of detector/transformer plugins and determining which of the at least the portion of the plurality of detector/transformer plugins is capable of transforming the updated source electronic data file into the updated transformed electronic data file;
performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the selected detector/transformer plugin; and
providing the updated transformed electronic data file for importing into the data analysis system.

16. The non-transitory computer-readable storage medium of claim 15 , wher

Assignees

• Palantir Technologies Inc

Inventors

• Yazicioglu Stephen
• Luck Christopher
• Giardina Robert
• Streufert Justin
• Slatcher Timothy
• O'Connor Gregory
• Marc-Aurele Brandon
• Zhu Olivia
• Schindel Howard
• Tung Henry
• Ray Lucas
• Leech Christopher
• Jeney Eric
• Negrus Stefan
• Lee Jason
• Mingione Alessandro
• Doyle John McKinstry
• Pitelka Hunter
• Lozano Ethan
• Ossher Joel
• Fedderly Matthew

Classifications

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title

• G06F16/86

  Mapping to a database · CPC title

• G06F16/235

  Update request formulation · CPC title

• G06F16/162

  Delete operations (erasing in storage systems G06F3/0652) · CPC title