Indexing and relaying data to hot storage

Patent metadata

Field	Value
Publication number	US-12481635-B2
Application number	US-202418440724-A
Country	US
Kind code	B2
Filing date	Feb 13, 2024
Priority date	May 9, 2018
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

A method comprises receiving, from one or more search nodes of a distributed system, one or more requests for log data, the one or more search nodes being associated with one or more first storage systems; identifying an indexed portion of the log data stored in a second storage system of one or more second storage systems based on a particular request of the one or more requests, the log data being indexed by one or more indexing nodes independently from the one or more requests being received and processed by the one or more search nodes; changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.

First claim

What is claimed is:

1 . A method of processing search requests for log data, comprising:
receiving, from one or more search nodes of a distributed system, one or more requests for log data, the one or more search nodes being associated with one or more first storage systems local to the one or more search nodes; identifying, from an index catalog, an indexed portion of the log data and a location where the indexed portion is stored within a second storage system of one or more second storage systems separate from the one or more first storage systems based on a particular request of the one or more requests, the log data being indexed by one or more indexing nodes independently from the one or more requests being received and processed by the one or more search nodes; changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently, wherein the method is performed using one or more processors.

2 . The method of claim 1 , further comprising sending the indexed portion to at least one search node of the one or more search nodes, in response to the particular request, for storage in the associated at least one first storage system.

3 . The method of claim 2 , further comprising:
tracking availability of the indexed portion of the log data in the at least one first storage system; receiving a new request for the log data; serving the new request with a part of the log data in the at least one first storage system.

4 . The method of claim 2 , the sending comprising:
fetching the indexed portion from the second storage system; de-duplicating the indexed portion.

5 . The method of claim 2 , the changing being performed based on a number of requests for the log data received over a period of time or an amount of time for which indexed portions have been stored at the at least one first storage system.

6 . The method of claim 1 , the one or more first storage systems offering quicker data access than the one or more second storage systems.

7 . The method of claim 1 , the particular request identifying timing of the log data, a type of the log data, an application, a system, or a server.

8 . The method of claim 1 , the updating being performed based on an amount or a streaming rate of the log data.

9 . The method of claim 1 , the log data being immutable to the one or more search nodes.

10 . The method of claim 1 , the receiving comprising accepting, via the one or more search nodes, the one or more requests for log data from an aggregator node that exposes a remote procedure call application programming interface.

11 . A computer system for processing search requests for log data, comprising:
a memory; one or more processors coupled to the memory and configured to perform: receiving, from one or more search nodes of a distributed system, one or more requests for log data, the one or more search nodes being associated with one or more first storage systems local to the one or more search nodes; identifying, from an index catalog, an indexed portion of the log data and a location where the indexed portion is stored within a second storage system of one or more second storage systems separate from the one or more first storage systems based on a particular request of the one or more requests, the log data being indexed by one or more indexing nodes independently from the one or more requests being received and processed by the one or more search nodes; changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.

12 . The computer system of claim 11 , the one or more processors further configured to perform sending the indexed portion to at least one search node of the one or more search nodes, in response to the particular request, for storage in the associated at least one first storage system.

13 . The computer system of claim 12 , the one or more processors further configured to perform:
tracking availability of the indexed portion of the log data in the at least one first storage system; receiving a new request for the log data; serving the new request with a part of the log data in the at least one first storage system.

14 . The computer system of claim 12 , the sending comprising:
fetching the indexed portion from the second storage system; de-duplicating the indexed portion.

15 . The computer system of claim 12 , the changing being performed based on a number of requests for the log data received over a period of time or an amount of time for which indexed portions have been stored at the at least one first storage system.

16 . The computer system of claim 11 , the one or more first storage systems offering quicker data access than the one or more second storage systems.

17 . The computer system of claim 11 , the particular request identifying timing of the log data, a type of the log data, an application, a system, or a server.

18 . The computer system of claim 11 , the updating being performed based on an amount or a streaming rate of the log data.

19 . The computer system of claim 11 , the log data being immutable to the one or more search nodes.

20 . The computer system of claim 11 , the receiving comprising accepting, via the one or more search nodes, the one or more requests for log data from an aggregator node that exposes a remote procedure call application programming interface.

Assignees

• Palantir Technologies Inc

Inventors

• Fink Robert
• Mallah Amr Al
• Turki Haithem

Classifications

• G06F16/212

  with details for data modelling support · CPC title

• G06F16/2228

  Indexing structures · CPC title

• G06F16/245

  Query processing · CPC title

• G06F16/2379

  Updates performed during online database operations; commit processing · CPC title

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title