Reassigning processing tasks to an external storage system

Patent metadata

Field	Value
Publication number	US-12248484-B2
Application number	US-201916657894-A
Country	US
Kind code	B2
Filing date	Oct 18, 2019
Priority date	Jul 31, 2017
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

Systems and methods are described for reducing execution time of a query that references external data systems. The system can determine an external data system is capable of processing one or more map or reduce phases of a map-reduce operation. When it is determined that the external data system can process a map or reduce phase, associated operations may be reassigned from the system to the external data system reducing the processing resources used by the system to response to the query and, in some cases, speeding up performance of the query.

First claim

What is claimed:

1. A method, comprising:
receiving, at a data intake and query system comprising one or more hardware processors, a query;
determining, by accessing a configuration file, that the query references an external search for a remote data store that stores data referenced by the external search, wherein the external search comprises a subquery of the query that is identified from the configuration file based at least in part on a reference within the query;
determining that the remote data store includes processing capabilities;
determining a type of processing capabilities of the remote data store;
identifying additional processing to be performed on results of the external search based on the query;
determining an amount of time to perform the additional processing at the remote data store;
determining that the amount of time to perform the additional processing at the remote data store does not exceed a threshold time, wherein the threshold time is based at least in part on an amount of time to perform the additional processing at the data intake and query system, and wherein the threshold time is based at least in part on an amount of computing resources available at the data intake and query system to execute the additional processing; and
in response to determining that the amount of time to perform the additional processing at the remote data store does not exceed the threshold time,
generating additional processing instructions based on the identified additional processing to be performed on the results of the external search and the determined type of processing capabilities of the remote data store; and
communicating the external search in combination with the additional processing instructions to the remote data store to perform preprocessing of search results of the external search to obtain a modified result of the external search to be received from the remote data store in place of the search results, wherein the amount of time to perform the additional processing at the remote data store further comprises an amount of time to transmit the modified result of the external search to the data intake and query system, and wherein the modified result of the external search comprises a reduced data set and wherein the amount of time to transmit the modified result of the external search accounts for a reduced transmission time to transmit the reduced data set.

2. The method of claim 1 , wherein determining that the remote data store includes processing capabilities comprises determining that the remote data store exposes at least some of the processing capabilities for access by an external system.

3. The method of claim 1 , wherein determining the type of processing capabilities of the remote data store further comprises determining whether the remote data store is capable of performing a map operation, a reduce operation, or both the map operation and the reduce operation in a map-reduce processing model.

4. The method of claim 1 , wherein determining the type of processing capabilities of the remote data store further comprises determining whether the remote data store can perform a set of operations in a map-reduce processing model without causing processing time to exceed the threshold time.

5. The method of claim 1 , wherein the configuration file stores an identity of a set of queries performable at one or more remote systems.

6. The method of claim 1 , wherein the additional processing instructions comprise instructions to perform a reduce operation when it is determined that the type of processing capabilities includes the capability of performing map operations and reduce operations.

7. The method of claim 1 , wherein the additional processing instructions comprise instructions to perform a map operation of a map-reduce processing model while omitting a reduce operation of the map-reduce processing model when it is determined that the type of processing capabilities includes the capability of performing map operations and not reduce operations.

8. The method of claim 1 , wherein, when determining that the type of processing capabilities of the remote data store includes the capability to perform reduce operations, the method further comprises:
determining that executing the query includes performing a reduce operation with respect to data stored at the remote data store; and
determining a processing time to perform the reduce operation at the remote data store.

9. The method of claim 8 , wherein the additional processing instructions comprise instructions to perform the reduce operation at the remote data store when it is determined that the processing time at the remote data store does not exceed a threshold time.

10. The method of claim 8 , wherein the additional processing instructions omits instructions to perform the reduce operation at the remote data store when it is determined that the processing time at the remote data store exceeds a threshold time.

11. The method of claim 8 , wherein determining the processing time to perform the reduce operation comprises estimating the processing time based at least in part on an estimate of the size of the data at the remote data store.

12. The method of claim 8 , wherein the reduce operation comprises a filtering operation or an aggregation operation.

13. The method of claim 1 , wherein communicating the external search in combination with the additional processing instructions comprises transmitting the external search in combination with the additional processing instructions to the remote data store to be executed by the remote data store.

14. The method of claim 1 , wherein communicating the external search in combination with the additional processing instructions comprises transmitting the external search in combination with the additional processing instructions to a worker node comprising one or more processors configured to interact with the remote data store.

15. The method of claim 1 , wherein at least a portion of the query is executed at a data intake and query system and wherein communicating the additional processing instructions to the remote data store or a worker node reduces the processing performed at the data intake and query system.

16. The method of claim 1 , wherein at least a portion of the query is executed at a data intake and query system and wherein the method further comprises modifying the at least the portion of the query executed at the data intake and query system based at least in part on the additional processing instructions.

17. The method of claim 16 , wherein modifying the at least the portion of the query executed at the data intake and query system reduces the operations performed at the data intake and query system.

18. The method of claim 1 , wherein the query references a second external search for a second remote data store.

19. The method of claim 18 , wherein the method further comprises:
determining that the second remote data store does not include processing capabilities; and
directing a worker node to perform the second external search with respect to the second remote data store without generating additional processing instructions to be performed by the second remote data store.

20. The method of claim 18 , wherein the method further comprises:
determining that generating a response to the query includes performing a reduce operation on a combination of a result of the external search and a result of the second external search;
determining that the reduce operation references d

Assignees

• Splunk Inc

Inventors

• Pal Sourav
• Bhattacharjee Arindam
• Patterson Wayne

Classifications

• G06F16/24549

  Run-time optimisation · CPC title

• G06F16/2471 Primary

  Distributed queries · CPC title

• G06F16/24545 Primary

  Selectivity estimation or determination · CPC title