Data analytic systems

Patent metadata

Field	Value
Publication number	US-11789912-B2
Application number	US-202217833173-A
Country	US
Kind code	B2
Filing date	Jun 6, 2022
Priority date	Oct 17, 2017
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

A method is disclosed, comprising receiving, at a build service, an external dataset and an adaptor associated with the external dataset, the external dataset being in a format which is not managed by the build server. The method may also comprise querying the adaptor, by the build service, for information related to the external dataset. The method may also comprise receiving, at the build service, the information related to the external dataset; and causing, by the build service, the external dataset to be input into a build pipeline using the information related to the external dataset.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving an external dataset and an adaptor associated with the external dataset, the external dataset being in a format which is not managed by a build service;
querying the adaptor for information related to the external dataset, wherein the information includes a version of data used in a previous build, a version of transformation code used to run the previous build, runtime parameters for the previous build, or a data schema used in the previous build;
receiving, by a processor, a build request at the build service to perform a first data build on the external dataset;
determining, by the processor, that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining whether the first data build is compatible with the second data build based on the querying,
wherein the first data build is determined to be incompatible with the second data build when the first data build and the second data build read and/or write to the same data objects with different input data versions, code versions, or build parameters; and
when the first data build is compatible with the second data build, causing the first and second data builds to be executed at least partially concurrently.

2. The computer-implemented method of claim 1 , further comprising: receiving a build request at the build service to perform a third data build; determining whether the third data build is scheduled to be at least partially contemporaneous with a fourth data build; in response to a positive determination, running the third data build; in response to a negative determination, deciding whether the third data build is compatible with the fourth data build.

3. The computer-implemented method of claim 2 , further comprising: in response to a positive decision, causing the third data build and the fourth data build to run concurrently; in response to a negative decision, causing the third data build and the fourth data build to run consecutively.

4. The computer-implemented method of claim 1 , wherein the first data build is determined to be compatible with the second data build when the first data build and the second data build read and/or write the same data object to the same dataset.

5. The computer-implemented method of claim 1 , wherein when the first data build is compatible with the second data build, requesting a user input as to whether to cause the first and second data builds to be executed at least partially concurrently, and in response to a positive input, causing the first and second data builds to be executed at least partially concurrently.

6. The computer-implemented method of claim 5 further comprising, in response to a negative user input, causing the first and second data builds to be executed consecutively.

7. The computer-implemented method of claim 5 , wherein causing the first and second data builds to be executed consecutively comprises determining which of the first and second data builds has priority over the other data build.

8. The computer-implemented method of claim 7 , wherein the priority is determined by which data build was initiated first.

9. The computer-implemented method of claim 7 , wherein the priority is determined based on a first user initiating the first data build and a second user initiating the second data build.

10. A computer system, comprising:
a memory; and one or more processors coupled to the memory and configured to perform:
receiving an external dataset and an adaptor associated with the external dataset, the external dataset being in a format which is not managed by a build service;
querying the adaptor for information related to the external dataset, wherein the information includes a version of data used in a previous build, a version of transformation code used to run the previous build, runtime parameters for the previous build, or a data schema used in the previous build;
receiving a build request at a build service to perform a first data build on the external dataset;
determining that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining whether the first data build is compatible with the second data build based on the querying, wherein the first data build is determined to be incompatible with the second data build when the first data build and the second data build read and/or write to the same data objects with different input data versions, code versions, or build parameters; and
when the first data build is compatible with the second data build, causing the first and second data builds to be executed at least partially concurrently.

11. One or more non-transitory computer-readable storage media storing instructions which when executed cause one or more processors to perform a method, the method comprising:
receiving an external dataset and an adaptor associated with the external dataset, the external dataset being in a format which is not managed by a build service;
querying the adaptor for information related to the external dataset, wherein the information includes a version of data used in a previous build, a version of transformation code used to run the previous build, runtime parameters for the previous build, or a data schema used in the previous build;
receiving a build request at the build service to perform a first data build on the external dataset;
determining that the first data build is scheduled to be at least partially contemporaneous with a second data build on a second dataset;
determining whether the first data build is compatible with the second data build based on the querying, wherein the first data build is determined to be incompatible with the second data build when the first data build and the second data build read and/or write to the same data objects with different input data versions, code versions, or build parameters; and
when the first data build is compatible with the second data build, causing the first and second data builds to be executed at least partially concurrently.

12. The one or more non-transitory computer-readable storage media of claim 11 , the method further comprising: receiving a build request at the build service to perform a third data build; determining whether the third data build is scheduled to be at least partially contemporaneous with a fourth data build; in response to a positive determination, running the third data build; in response to a negative determination, deciding whether the third data build is compatible with the fourth data build.

13. The one or more non-transitory computer-readable storage media of claim 12 , the method further comprising: in response to a positive decision, causing the third data build and the fourth data build to run concurrently; in response to a negative decision, causing the third data build and the fourth data build to run consecutively.

14. The one or more non-transitory computer-readable storage media of claim 11 , wherein the first data build is determined to be compatible with the second data build when the first data build and the second data build read and/or write the same data object to the same dataset.

15. The one or more non-transitory computer-readable storage media of claim 11 , wherein when the first data build is compatible with the second data build, requesting a user input as to whether to cause the first and second data builds to be executed at least partially concurrently, and in response to a positive input, causing the first and second data builds

Assignees

• Palantir Technologies Inc

Inventors

• Koenig Patrick
• Nepywoda Paul
• Fink Robert

Classifications

• G06F16/211 Primary

  Schema design and management · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/2455

  Query execution · CPC title

• G06F16/24532 Primary

  of parallel queries · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title