Indexing and relaying data to hot storage

Patent metadata

Field	Value
Publication number	US-11914566-B2
Application number	US-202318138492-A
Country	US
Kind code	B2
Filing date	Apr 24, 2023
Priority date	May 9, 2018
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

A method is disclosed. The method comprises receiving, from one or more search nodes of a distributed system, one or more requests for log data, the one or more search nodes being associated with one or more hot storage systems; identifying, from an index catalog, an indexed portion of the log data stored in a cold storage system of one or more cold storage systems based on at least part of the particular request, the index catalog containing pointers to indexed portions of the log data in the one or more cold storage systems, the indexing being performed by one or more indexing nodes independently from the receiving by the one or more search nodes; and sending the indexed portion to the one or more search nodes for storage in the associated one or more hot storage systems, wherein the method is performed using one or more processors.

First claim

The invention claimed is:

1. A method comprising:
receiving, from one or more search nodes of a distributed system, one or more requests for log data,
the one or more search nodes being associated with one or more hot storage systems;
identifying, from an index catalog, an indexed portion of the log data stored in a cold storage system of one or more cold storage systems based on a particular request of the one or more requests,
the index catalog containing pointers to indexed portions of the log data in the one or more cold storage systems,
the log data being indexed by one or more indexing nodes independently from the receiving by the one or more search nodes;
sending the indexed portion to at least one search nodes of the one or more search nodes in response to the particular request for storage in the associated at least one hot storage systems,
wherein the method is performed using one or more processors.

2. The method of claim 1 , further comprising:
tracking availability of the one or more search nodes;
determining the at least one search nodes based on the availability of the one or more search nodes.

3. The method of claim 1 , the at least one search nodes comprising multiple search nodes.

4. The method of claim 1 , the particular request identifying timing of the log data, a type of the log data, an application, a system, or a server.

5. The method of claim 1 , the sending comprising fetching the indexed portion from the cold storage system.

6. The method of claim 1 , further comprising:
tracking availability of the indexed portion of the log data in the at least one hot storage systems;
receiving a new request for the log data;
serving the new request with a part of the log data in the at least one hot storage systems.

7. The method of claim 1 , the log data being immutable to the one or more search nodes.

8. The method of claim 1 , further comprising changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.

9. The method of claim 8 , the updating being performed based on an amount or a streaming rate of the log data.

10. The method of claim 8 , the changing being performed based on a number of requests for the log data received over a period of time or an amount of time for which indexed portions have been stored at the one or more search nodes.

11. One or more non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed cause one or more processors to perform a method, the method comprising:
receiving, from one or more search nodes of a distributed system, one or more requests for log data,
the one or more search nodes being associated with one or more hot storage systems;
identifying, from an index catalog, an indexed portion of the log data stored in a cold storage system of one or more cold storage systems based on a particular request of the one or more requests,
the index catalog containing pointers to indexed portions of the log data in the one or more cold storage systems,
the log data being indexed by one or more indexing nodes independently from the receiving by the one or more search nodes;
sending the indexed portion to at least one search nodes of the one or more search nodes in response to the particular request for storage in the associated at least one hot storage systems.

12. The one or more non-transitory computer-readable storage media of claim 11 , the method further comprising:
tracking availability of the one or more search nodes;
determining the at least one search nodes based on the availability of the one or more search nodes.

13. The one or more non-transitory computer-readable storage media of claim 11 , the at least one search nodes comprising multiple search nodes.

14. The one or more non-transitory computer-readable storage media of claim 11 , the particular request identifying timing of the log data, a type of the log data, an application, a system, or a server.

15. The one or more non-transitory computer-readable storage media of claim 11 , the sending comprising fetching the indexed portion from the cold storage system.

16. The one or more non-transitory computer-readable storage media of claim 11 , the method further comprising:
tracking availability of the indexed portion of the log data in the at least one hot storage systems;
receiving a new request for the log data;
serving the new request with a part of the log data in the at least one hot storage systems.

17. The one or more non-transitory computer-readable storage media of claim 11 , the log data being immutable to the one or more search nodes.

18. The one or more non-transitory computer-readable storage media of claim 11 , the method further comprising changing a first number of the one or more search nodes and updating a second number of the one or more indexing nodes independently.

19. The one or more non-transitory computer-readable storage media of claim 18 , the updating being performed based on an amount or a streaming rate of the log data.

20. The one or more non-transitory computer-readable storage media of claim 18 , the changing being performed based on a number of requests for the log data received over a period of time or an amount of time for which indexed portions have been stored at the one or more search nodes.

Assignees

• Palantir Technologies Inc

Inventors

• Fink Robert
• Mallah Amr Al
• Turki Haithem

Classifications

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F16/2272 Primary

  Management thereof · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• G06F16/2228

  Indexing structures · CPC title

• G06F16/2379

  Updates performed during online database operations; commit processing · CPC title