Providing external access to a processing platform

Patent metadata

Field	Value
Publication number	US-11809216-B2
Application number	US-202217820062-A
Country	US
Kind code	B2
Filing date	Aug 16, 2022
Priority date	Nov 22, 2018
Publication date	Nov 7, 2023
Grant date	Nov 7, 2023

Abstract

An apparatus, and a method, performed by one or more processors are disclosed. The method receives a build request associated with performing an external data processing task on a first data set, the first data set being stored in memory associated with a data processing platform to be performed at a system external to the data processing platform. The method generates a task identifier for the data processing task, and providing, in association with the task identifier, the first data set to an agent associated with the external system with an indication of the data processing task, the agent being arranged to cause performance of the task at the external system, to receive a second data set resulting from performance of the task, and to provide the second data set and associated metadata indicative of the transformation. The method receives the second data set and metadata from the agent associated with the external system and stores the second data set and associated metadata.

First claim

What is claimed is:

1. A method performed by one or more processors of a data processing platform, the method comprising:
receiving a request associated with performing a data processing task on a first dataset at an external system;
generating a task identifier for the data processing task; and
providing, in association with the task identifier, the first dataset and an indication of the data processing task to an agent associated with the external system, the agent being arranged to cause performance of the data processing task at the external system.

2. The method of claim 1 , wherein the first dataset is stored in memory associated with the data processing platform, and wherein the agent is further arranged to:
receive a second dataset resulting from performance of the data processing task, and
provide the second dataset and associated metadata to the data processing platform.

3. The method of claim 2 , further comprising:
receiving the second dataset and associated metadata from the agent associated with the external system; and
storing the second dataset and associated metadata in the data processing platform.

4. The method of claim 3 , further comprising storing the task identifier for the request in a co-ordinator function that receives one or more build queries from the external system, or a service associated with the external system, for providing the first dataset and the indication of the data processing task to the agent.

5. The method of claim 4 , wherein the co-ordinator function stores multiple task identifiers for multiple requests for providing one or more requests to the agent.

6. The method of claim 5 , wherein the co-ordinator function removes task identifiers, or flags task identifiers as used, responsive to receiving a completion message from the agent, such that subsequent build queries from the same external system do not cause re-provision of the same datasets.

7. The method of claim 6 , further comprising generating a token associated with the request for enabling verification at the data processing platform that the received second dataset corresponds to the request.

8. The method of claim 7 , wherein the metadata comprises one or more of: at least part of a code used on the external system to perform the data processing task; a version number of the code used on the external system to perform the data processing task; the date the code was last updated; or a hash of the code.

9. The method of claim 8 , further comprising transmitting an update query to the external system, and responsive thereto, receiving an indication of one or more updates made to code used on the external system to perform the data processing task.

10. The method of claim 9 , further comprising, responsive to receiving an indication of one or more updates made to the code on the external system, flagging the second dataset, and/or any other dataset generated as a result of performance of the data processing task, as being out of date.

11. The method of claim 10 , further comprising displaying a prompt on a user interface indicating the second and/or other datasets flagged as out of date.

12. The method of claim 11 , further comprising preventing further data processing tasks being performed on the second and/or other datasets flagged as out of date.

13. The method of claim 12 , further comprising automatically generating a new request for performing the data processing task on the first dataset for generating a new dataset to replace the second and/or other datasets flagged as out of date.

14. The method of claim 13 , wherein the data processing task comprises one task of a data processing pipeline comprising two or more tasks, and wherein the received second dataset is provided as input to a further task which performs a further data processing task to generate a third dataset.

15. A data processing platform comprising:
one or more processors configured to execute program instructions to perform the method of claim 1 .

Assignees

• Palantir Technologies Inc

Inventors

• Kuan Audrey
• Kaier Andrew
• Lee Eric
• Grewal Jasjit
• Elliot Mark
• Kulkarni Nitish
• Fink Robert
• Rogerson Samuel
• Pearson Thomas
• Powell Thomas
• Manning Lawrence
• Garvey Corey

Classifications

• G06F8/71 Primary

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F9/3885

  using a plurality of independent parallel functional units · CPC title

• G06F9/4494

  data driven · CPC title

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F16/2329

  using versioning · CPC title