Universal data pipeline

Patent metadata

Field	Value
Publication number	US-9946738-B2
Application number	US-201615287715-A
Country	US
Kind code	B2
Filing date	Oct 6, 2016
Priority date	Nov 5, 2014
Publication date	Apr 17, 2018
Grant date	Apr 17, 2018

Abstract

A history preserving data pipeline computer system and method. In one aspect, the history preserving data pipeline system provides immutable and versioned datasets. Because datasets are immutable and versioned, the system makes it possible to determine the data in a dataset at a point in time in the past, even if that data is no longer in the current version of the dataset.

First claim

The invention claimed is:

1. A method comprising:
at one or more computing devices comprising one or more processors and one or more storage media storing one or more computer programs executed by the one or more processors to perform the method, performing operations comprising:
maintaining a build catalog comprising a plurality of build catalog entries;
wherein each build catalog entry, of the plurality of build catalog entries, comprises:
an identifier of a version of a derived dataset corresponding to the build catalog entry,
one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and
a derivation program build dependency of the version of the derived dataset corresponding to the build catalog entry, the derivation program build dependency comprising an identifier of a version of a derivation program executed to generate the version of the derived dataset corresponding to the build catalog entry;

creating a new version of a particular derived dataset in context of a successful transaction;
adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of the new version of the particular derived dataset, the identifier of the new version of the particular derived dataset being a transaction commit identifier assigned to the successful transaction;
wherein the creating the new version of the particular derived dataset is based on executing a particular version of a particular derivation program;
wherein the new build catalog entry comprises an identifier of the particular version of the particular derivation program;
wherein the creating the new version of the particular derived dataset is based on providing one or more particular child dataset versions as input to the executing the particular version of the particular derivation program; and
wherein the new build catalog entry comprises an identifier of each of the one or more particular child dataset versions.

2. One or more non-transitory storage media storing one or more computer programs, the one or more computer programs comprising instructions for performing operations comprising:
maintaining a build catalog comprising a plurality of build catalog entries;
wherein each build catalog entry, of the plurality of build catalog entries, comprises:
an identifier of a version of a derived dataset corresponding to the build catalog entry,
one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and
a derivation program build dependency of the version of the derived dataset corresponding to the build catalog entry, the derivation program build dependency comprising an identifier of a version of a derivation program executed to generate the version of the derived dataset corresponding to the build catalog entry;

creating a new version of a particular derived dataset in context of a successful transaction; and
adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of the new version of the particular derived dataset, the identifier of the new version of the particular derived dataset being a transaction commit identifier assigned to the successful transaction;
wherein the creating the new version of the particular derived dataset is based on executing a particular version of a particular derivation program;
wherein the new build catalog entry comprises an identifier of the particular version of the particular derivation program;
wherein the creating the new version of the particular derived dataset is based on providing one or more particular child dataset versions as input to the executing the particular version of the particular derivation program; and
wherein the new build catalog entry comprises an identifier of each of the one or more particular child dataset versions.

3. A system comprising:
one or more hardware processors;
one or more computer programs; and
one or more storage media storing the one or more computer programs for execution by the one or more hardware processors, the one or more computer programs comprising instructions for performing operations comprising:
maintaining a build catalog comprising a plurality of build catalog entries;
wherein each build catalog entry, of the plurality of build catalog entries, comprises:
an identifier of a version of a derived dataset corresponding to the build catalog entry,
one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and
a derivation program build dependency of the version of the derived dataset corresponding to the build catalog entry, the derivation program build dependency comprising an identifier of a version of a derivation program executed to generate the version of the derived dataset corresponding to the build catalog entry;

creating a new version of a particular derived dataset in context of a successful transaction;
adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of the new version of the particular derived dataset, the identifier of the new version of the particular derived dataset being a transaction commit identifier assigned to the successful transaction;
wherein the creating the new version of the particular derived dataset is based on executing a particular version of a particular derivation program;
wherein the new build catalog entry comprises an identifier of the particular version of the particular derivation program;
wherein the creating the new version of the particular derived dataset is based on providing one or more particular child dataset versions as input to the executing the particular version of the particular derivation program; and
wherein the new build catalog entry comprises an identifier of each of the one or more particular child dataset versions.

Assignees

• Palantir Technologies Inc

Inventors

• Meacham Jacob
• Harris Michael
• Brodman Gustav
• Cuthriell Lynn
• Korus Hannah
• Toth Brian
• Hsiao Jonathan
• Elliot Mark
• Schimpf Brian
• Garland Michael
• Nguyen Evelyn

Classifications

• G06F17/3023

  Physics · mapped topic

• G06F17/30227

  Physics · mapped topic

• G06F17/30309 Primary

  Physics · mapped topic

• G06F17/30292

  Physics · mapped topic

• G06F17/30371

  Physics · mapped topic