Ingesting data to a processing platform

Patent metadata

Field	Value
Publication number	US-11630752-B2
Application number	US-202016893940-A
Country	US
Kind code	B2
Filing date	Jun 5, 2020
Priority date	Jul 12, 2019
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

Example embodiments describe a method performed by one or more processors. The method may comprise sending over a network, to a software component installed at a remote data source, a request to download data stored at, or in association with, the remote data source, the software component being configured to access performance data at said remote data source. In response to sending the request, the method may comprise receiving from the software component at least an indication of the performance data accessed by said software component, determining whether to proceed with the data download request or to modify the data download request based on the received performance data.

First claim

What is claimed is:

1. A method, performed by one or more processors, comprising:
sending over a network, to a software component installed at remote data source, a request to download data stored at, or in association with, the remote data source, the software component being configured to access performance data at said remote data source;
in response to sending the request, receiving from the software component at least an indication of the performance data accessed by said software component, wherein the performance data is associated with one or more application servers executing on the remote data source and configured to provide interactivity to one or more users of the remote data source;
determining whether to proceed with the data download request or to pause the data download request for a first predetermined period of time based on the received indication of the performance data; and
in response to determining to pause the data download request for the first predetermined period of time:
re-sending the data download request after the first predetermined period of time; and
determining whether to proceed with the re-sent data download request or to pause the re-sent data download request for a second predetermined period of time based on a received indication of up-to-date performance data.

2. The method of claim 1 , wherein the second predetermined period of time is increased as compared to the first predetermined period of time.

3. The method of claim 1 , wherein the accessed performance data comprises performance metrics comprising one or more of RAM usage, CPU usage, process usage, or number of users.

4. The method of claim 3 , wherein determining whether to proceed with or modify the data download request comprises accessing one or more thresholds representing one or more of the performance metrics against which to test the received indication of the performance data.

5. The method of claim 4 , further comprising sending an initialisation request to the software component, the initialisation request being a paging request for requesting the data source to build a page comprising N data objects, N being user-configurable, wherein the download request is sent subsequently, and wherein proceeding with the data download request is only performed if the page comprising the N data objects has been built.

6. The method of claim 5 , wherein a message is returned from the software component if the page comprising the N data objects has not been built, and responsive to said message, the data download request is paused and the data download request re-sent after a predetermined time period.

7. The method of claim 6 , further comprising receiving a requested page comprising N data objects from the software component and sending, in response to receiving the page, an acknowledgment message to the software component for causing another page comprising a different set of N data objects to be built.

8. The method of claim 4 , further comprising sending to the software component a unique identifier associated with the download request and one or more subsequent download requests, wherein the unique identifier is used by the software component to recognise that download requests are associated with a common target system, and responsive thereto, receiving incremental updates made at the data source made subsequent to a previous request with the same unique identifier.

9. The method of claim 8 , further comprising providing access at a target system, via the software component, to one or more application-layer resources made available to the source system.

10. A method, performed by one or more processors, comprising:
receiving by a software component, installed at a data source from a remote target, a request to download data stored at, or associated with, the data source;
determining, by the software component, performance data of the data source, wherein the performance data is associated with one or more application servers executing on the data source and configured to provide interactivity to one or more users of the data source;
accessing, by the software component, performance data of the data source;
in response to receiving the request, sending from the software component to the remote target at least an indication of the performance data accessed by the software component;
proceeding with the data download request only in response to a further request from the remote target;
re-receiving the data download request after a predetermined time period; and
in response to re-receiving the request, sending from the software component to the remote target at least an indication of up-to-date performance data accessed by said software component.

11. The method of claim 10 , wherein the predetermined time period is determined by the remote target, and wherein the predetermined time period increases for a subsequent download request.

12. The method of claim 10 , wherein the accessed performance data comprises performance metrics comprising one or more of RAM usage, CPU usage, process usage, or number of users, and wherein the software component determines which of the performance metrics to send to the remote target based on configuration data received therefrom.

13. The method of claim 12 , further comprising receiving an initialisation request by the software component, the initialisation request being a paging request for requesting the data source to build a page comprising N data objects, N being user-configurable, wherein the download request is received subsequently, and a message is subsequently returned to the remote target responsive to the download request if the page comprising N data objects has not been built.

14. The method of claim 12 , further comprising receiving by the software component a unique identifier associated with the download request and one or more subsequent download requests, wherein the unique identifier is used by the software component to recognise that download requests are associated with a common target system, and responsive thereto, sending to the remove target incremental updates made at the data source made subsequent to a previous request with the same unique identifier.

15. The method of claim 14 , further comprising providing access to a target system, via the software component, to one or more application-layer resources made available to the source system.

Assignees

• Palantir Technologies Inc

Inventors

• Sezgin Cenk
• Grewal Jasjit
• Pearson Thomas

Classifications

• H04L67/06

  specially adapted for file transfer, e.g. file transfer protocol [FTP] · CPC title

• G06F9/5072

  Grid computing · CPC title

• H04L67/60

  Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70) · CPC title

• G06F11/3466 Primary

  Performance evaluation by tracing or monitoring · CPC title

• G06F9/547 Primary

  Remote procedure calls [RPC]; Web services · CPC title