Ingesting data to a processing platform

Patent metadata

Field	Value
Publication number	US-12339761-B2
Application number	US-202318183047-A
Country	US
Kind code	B2
Filing date	Mar 13, 2023
Priority date	Jul 12, 2019
Publication date	Jun 24, 2025
Grant date	Jun 24, 2025

Abstract

Example embodiments describe a method performed by one or more processors. The method may comprise sending over a network, to a software component installed at a remote data source, a request to download data stored at, or in association with, the remote data source, the software component being configured to access performance data at said remote data source. In response to sending the request, the method may comprise receiving from the software component at least an indication of the performance data accessed by said software component, determining whether to proceed with the data download request or to modify the data download request based on the received performance data.

First claim

What is claimed is:

1. A method, performed by one or more processors, comprising:
receiving, by a target device, and from a software component installed at a remote data source, at least an indication of performance data accessed by said software component, wherein the performance data is associated with one or more application servers executing on the remote data source and configured to run an application and provide interactivity to one or more users of the remote data source, and wherein the performance data comprises metrics of resource utilization on the remote data source running the application;
determining, by the target device, and based on the indication of the performance data received from the software component installed at the remote data source, either to proceed with a download of first data stored at the remote data source, or to modify the download;
in response to determining, based on the received indication of the performance data, to proceed with the download, proceeding, by the target device, to download the first data; and
in response to determining, based on the indication of the performance data received from the software component installed at the remote data source and running the application, to modify the download, proceeding, by the target device, with downloading second data stored at the remote data source, wherein the second data is different from the first data in that the second data comprises a different quantity of rows of data as compared to the first data, and wherein the second data comprises a different amount of data as compared to the first data.

2. The method of claim 1 further comprising:
in response to determining to modify the download, pausing the download for a first period of time.

3. The method of claim 2 further comprising:
after the first period of time, receiving, from the software component, at least an indication of updated performance data associated with the one or more application servers; and
determining, based on the received indication of the updated performance data, whether to proceed with the download or to pause the download for a second period of time.

4. The method of claim 3 , wherein the second period of time is increased as compared to the first period of time.

5. The method of claim 1 , wherein the performance data comprises performance metrics comprising one or more of RAM usage, CPU usage, process usage, or number of users.

6. The method of claim 5 , wherein determining whether to proceed with or modify the download comprises accessing one or more thresholds representing one or more of the performance metrics against which to test the received indication of the performance data.

7. The method of claim 1 further comprising:
sending an initialisation request to the software component, the initialisation request being a paging request for requesting the remote data source to build a page comprising N data objects, N being user-configurable, wherein a download request is sent subsequently, and wherein proceeding with the download is only performed if the page comprising the N data objects has been built.

8. The method of claim 7 , wherein a message is returned from the software component if the page comprising the N data objects has not been built, and responsive to said message, the download is paused and the download request is re-sent after a predetermined time period.

9. The method of claim 8 further comprising:
receiving a requested page comprising N data objects from the software component; and
sending, in response to receiving the page, an acknowledgment message to the software component for causing another page comprising a different set of N data objects to be built.

10. The method of claim 1 further comprising:
sending to the software component a unique identifier associated with the download and one or more subsequent download requests, wherein the unique identifier is used by the software component to recognize that download requests are associated with a common target system, and responsive thereto, receiving incremental updates made at the remote data source made subsequent to a previous request with the same unique identifier.

11. The method of claim 10 further comprising:
providing access at a target system, via the software component, to one or more application-layer resources made available to the remote data source.

12. A system comprising:
one or more computer readable storage mediums having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the system to perform the method of claim 1 .

13. A computer program product comprising one or more computer-readable storage mediums having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform the method of claim 1 .

Assignees

• Palantir Technologies Inc

Inventors

• Sezgin Cenk
• Grewal Jasjit
• Pearson Thomas

Classifications

• G06F11/3409

  for performance assessment · CPC title

• G06F11/302

  where the computing system component is a software system · CPC title

• G06F8/61

  Installation · CPC title

• H04L67/60

  Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70) · CPC title

• H04L67/06

  specially adapted for file transfer, e.g. file transfer protocol [FTP] · CPC title