Data analytic systems

Patent metadata

Field	Value
Publication number	US-12105682-B2
Application number	US-202318488826-A
Country	US
Kind code	B2
Filing date	Oct 17, 2023
Priority date	Oct 17, 2017
Publication date	Oct 1, 2024
Grant date	Oct 1, 2024

Abstract

A method comprises receiving, at a build service of a build server, an external dataset and an adaptor application module, the external dataset being in a specific format, the adaptor application module providing information relevant to a build pipeline maintained by the build service for building an output dataset based on the external dataset, the information including changes to the external dataset since a previous build of the output dataset is performed and a data schema used in the previous build, the build pipeline involving data only in one or more formats other than the specific format; incorporating the external dataset into the build pipeline without the external dataset being reformatted in accordance with requirements of the build service; receiving a request from the adaptor application module for specific information relating to a most recent data build run by the build service; providing a response to the adaptor application module.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, at a build service of a build server, an external dataset and an adaptor application module, the external dataset being in a specific format,
the adaptor application module providing information relevant to a build pipeline maintained by the build service for building an output dataset based, at least in part, on the external dataset,
the information including changes to the external dataset since a previous build of the output dataset is performed and a data schema used in the previous build,
the build pipeline involving data only in one or more formats other than the specific format;
incorporating the external dataset into the build pipeline without the external dataset being reformatted in accordance with requirements of the build service;
receiving a request from the adaptor application module for specific information relating to a most recent data build run by the build service;
providing a response to the adaptor application module.

2. The computer-implemented method of claim 1 , further comprising
querying the adaptor application module for data related to the external dataset,
the information being provided in response to the querying.

3. The computer-implemented method of claim 1 , the information including a title or other metadata of the external dataset.

4. The computer-implemented method of claim 1 , the information indicating a version of data used in the previous build or a version of transformation code used to run the previous build, runtime parameters for the previous build.

5. The computer-implemented method of claim 1 , further comprising:
receiving a build request at the build service to perform a first data build on the external dataset;
determining that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining, based on the information, that the first data build and the second data build are compatible in that an order of executing the first data build and the second data build does not affect outputs of the first data build and the second data build;
causing the first data build and the second data build to be executed at least partially concurrently.

6. The computer-implemented method of claim 1 , further comprising:
receiving a build request at the build service to perform a first data build on the external dataset;
determining that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining, based on the information, that the first data build and the second data build are incompatible in that an order of executing the first data build and the second data build affects outputs of the first data build and the second data build.

7. The computer-implemented method of claim 6 , the first data build and the second data build reading and/or writing to common data objects with different input data versions, code versions, or build parameters.

8. The computer-implemented method of claim 6 , further comprising causing the first data build and the second data build to be executed consecutively.

9. The computer-implemented method of claim 6 , further comprising:
determining priorities associated with the first data build and the second data build;
causing executing the first data build and the second data build based on the priorities.

10. The computer-implemented method of claim 1 , further comprising:
receiving a build request at the build service to perform a first data build on the external dataset;
determining that the first data build is not scheduled to be at least partially contemporaneous with a second data build on a second dataset;
causing executing the first data build.

11. One or more non-transitory, computer-readable storage media storing one or more sequences of instructions which, when executed cause one or more processors to perform a method, the method comprising:
receiving, at a build service of a build server, an external dataset and an adaptor application module, the external dataset being in a specific format,
the adaptor application module providing information relevant to a build pipeline maintained by the build service for building an output dataset based, at least in part, on the external dataset,
the information including changes to the external dataset since a previous build of the output dataset is performed and a data schema used in the previous build,
the build pipeline involving data only in one or more formats other than the specific format;
incorporating the external dataset into the build pipeline without the external dataset being reformatted in accordance with requirements of the build service;
receiving a request from the adaptor application module for specific information relating to a most recent data build run by the build service;
providing a response to the adaptor application module.

12. The one or more non-transitory, computer-readable storage media of claim 11 , the method further comprising
querying the adaptor application module for data related to the external dataset,
the information being provided in response to the querying.

13. The one or more non-transitory, computer-readable storage media of claim 11 , the information including a title or other metadata of the external dataset.

14. The one or more non-transitory, computer-readable storage media of claim 11 , the information indicating a version of data used in the previous build or a version of transformation code used to run the previous build, runtime parameters for the previous build.

15. The one or more non-transitory, computer-readable storage media of claim 11 , the method further comprising:
receiving a build request at the build service to perform a first data build on the external dataset;
determining that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining, based on the information, that the first data build and the second data build are compatible in that an order of executing the first data build and the second data build does not affect outputs of the first data build and the second data build;
causing the first data build and the second data build to be executed at least partially concurrently.

16. The one or more non-transitory, computer-readable storage media of claim 11 , the method further comprising:
receiving a build request at the build service to perform a first data build on the external dataset;
determining that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining, based on the information, that the first data build and the second data build are incompatible in that an order of executing the first data build and the second data build affects outputs of the first data build and the second data build.

17. The one or more non-transitory, computer-readable storage media of claim 16 , the first data build and the second data build reading and/or writing to common data objects with different input data versions, code versions, or build parameters.

18. The one or more non-transitory, computer-readable storage media of claim 16 , further comprising causing the first data build and the second data build to be executed consecutively.

19. The one or more non-transitory, computer-readable storage media of claim 16 , further comprising:
determining priorities associated wit

Assignees

• Palantir Technologies Inc

Inventors

• Koenig Patrick
• Nepywoda Paul
• Fink Robert

Classifications

• G06F16/24532 Primary

  of parallel queries · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/2455

  Query execution · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/211 Primary

  Schema design and management · CPC title