Providing external access to a processing platform

Patent metadata

Field	Value
Publication number	US-11449333-B2
Application number	US-202016900071-A
Country	US
Kind code	B2
Filing date	Jun 12, 2020
Priority date	Nov 22, 2018
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

An apparatus, and a method, performed by one or more processors are disclosed. The method may comprise receiving a build request associated with performing an external data processing task on a first data set, the first data set being stored in memory associated with a data processing platform to be performed at a system external to the data processing platform. The method may also comprise generating a task identifier for the data processing task, and providing, in association with the task identifier, the first data set to an agent associated with the external system with an indication of the data processing task, the agent being arranged to cause performance of the task at the external system, to receive a second data set resulting from performance of the task, and to provide the second data set and associated metadata indicative of the transformation. The method may also comprise receiving the second data set and metadata from the agent associated with the external system and storing the second data set and associated metadata.

First claim

What is claimed is:

1. A method performed by one or more processors configured to implement an agent of an external system, the method comprising:
identifying a request associated with performing a data processing task to be performed on a first dataset;
receiving, from a data processing platform, the first dataset and a task identifier for the data processing task to be performed on the first dataset, wherein the first dataset is transmitted from the data processing platform to the agent;
causing performance of the data processing task at the external system; and
transmitting an indication of one or more updates made to code to perform the data processing task.

2. The method of claim 1 further comprising:
receiving a second dataset resulting from performance of the data processing task; and
sending the second dataset and associated metadata to the data processing platform.

3. The method of claim 2 , wherein the first dataset is stored in memory associated with the data processing platform.

4. The method of claim 2 , further comprising transmitting one or more build queries to the data processing platform for causing receiving of the request,
wherein the one or more build queries are polling queries periodically transmitted to the data processing platform.

5. The method of claim 2 , further comprising receiving a token associated with the request for transmitting to the data processing platform when requesting the first dataset and/or sending the second dataset such that the data processing platform may determine that they correspond to the request.

6. The method of claim 1 , further comprising receiving an update query from the data processing platform, and responsive thereto, transmitting the indication of the one or more updates.

7. The method of claim 1 , further comprising, upon receiving the request, creating a directory for holding the first dataset and the second dataset and associated metadata, and removing the directory subsequent to the second dataset and associated metadata being sent to the data processing platform.

8. A computer system comprising:
one or more processors configured to execute program instructions to cause the computer system to implement an agent configured to:
identify a request associated with performing a data processing task to be performed on a first dataset;
receive, from a data processing platform, the first dataset and a task identifier for the data processing task to be performed on the first dataset, wherein the first dataset is transmitted from the data processing platform to the agent;
cause performance of the data processing task at an external system of the computer system; and
transmit an indication of one or more updates made to code to perform the data processing task.

9. The computer system of claim 8 , wherein the agent is further configured to:
receive a second dataset resulting from performance of the data processing task; and
send the second dataset and associated metadata to the data processing platform.

10. The computer system of claim 9 , wherein the first dataset is stored in memory associated with the data processing platform.

11. The computer system of claim 9 , wherein the one or more processors are configured to execute program instructions to further cause the computer system to implement the external system configured to:
receive the data processing task from the agent;
perform the data processing task;
generate the second dataset resulting from performance of the data processing task; and
provide the second dataset to the agent.

12. The computer system of claim 9 , wherein the agent is further configured to:
transmit one or more build queries to the data processing platform for causing receiving of the request,
wherein the one or more build queries are polling queries periodically transmitted to the data processing platform.

13. The computer system of claim 9 , wherein the agent is further configured to:
receive a token associated with the request for transmitting to the data processing platform when requesting the first dataset and/or sending the second dataset such that the data processing platform may determine that they correspond to the request.

14. The computer system of claim 8 , wherein the agent is further configured to:
receive an update query from the data processing platform, and responsive thereto, transmit the indication of the one or more updates.

15. The computer system of claim 8 , wherein the agent is further configured to:
upon receiving the request, create a directory for holding the first dataset and the second dataset and associated metadata; and
remove the directory subsequent to the second dataset and associated metadata being sent to the data processing platform.

Assignees

• Palantir Technologies Inc

Inventors

• Kuan Audrey
• Kaier Andrew
• Lee Eric
• Grewal Jasjit
• Elliot Mark
• Kulkarni Nitish
• Fink Robert
• Rogerson Samuel
• Pearson Thomas
• Powell Thomas
• Manning Lawrence
• Garvey Corey

Classifications

• G06F9/4494

  data driven · CPC title

• G06F16/2329

  using versioning · CPC title

• G06F16/2358

  Change logging, detection, and notification (replication G06F16/27) · CPC title

• G06F8/71 Primary

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title