Efficient data retrieval in big-data processing systems

Patent metadata

Field	Value
Publication number	US-10558680-B2
Application number	US-201715837681-A
Country	US
Kind code	B2
Filing date	Dec 11, 2017
Priority date	Jan 24, 2017
Publication date	Feb 11, 2020
Grant date	Feb 11, 2020

Abstract

A mechanism is provided for implementing operation optimization. Responsive to a request to load data via a input/output (I/O) load operation, prior to loading the data, transformed data in a local memory is searched for a match to the requested data. Responsive to identifying transformed that matches the requested data, the transformed data in the local memory is used to fulfill the request. Responsive to failing to identify transformed data in the local memory that matches the requested data, untransformed data in the local memory is searched for a match to the requested data. Responsive to identifying untransformed data that matches the requested data, the untransformed data in the local memory is used to fulfill the request. Responsive to failing to identify untransformed data in the local memory that matches the requested data, the requested data may be loaded from the storage system via the I/O load operation.

First claim

What is claimed is:

1. A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement operation optimization in the data processing system, the method comprising:
responsive to a request from an application to load data from a storage system via a input/output (I/O) load operation, prior to loading the data from the storage system, searching, by the processor, transformed data in a local memory to identify data that matches the requested data;
responsive to identifying transformed data in the local memory that matches the requested data, using, by the processor, the transformed data in the local memory to fulfill the request;
responsive to failing to identify transformed data in the local memory that matches the requested data, searching, by the processor, untransformed data in the local memory to identify data that matches the requested data;
responsive to identifying untransformed data in the local memory that matches the requested data, using, by the processor, the untransformed data in the local memory to fulfill the request; and
responsive to failing to identify untransformed data in the local memory that matches the requested data, loading, by the processor, the requested data from the storage system via the I/O load operation.

2. The method of claim 1 , further comprising:
responsive to failing to identify untransformed data in the local memory that matches the requested data and prior to loading the requested data from the storage system via the I/O load operation, searching, by the processor, transformed data in a shared memory to identify data that matches the requested data;
responsive to identifying transformed data in the shared memory that matches the requested data, using, by the processor, the transformed data in the shared memory to fulfill the request;
responsive to failing to identify transformed data in the shared memory that matches the requested data, searching, by the processor, untransformed data in the shared memory to identify data that matches the requested data;
responsive to identifying untransformed data in the shared memory that matches the requested data, using, by the processor, the untransformed data in the shared memory to fulfill the request; and
responsive to failing to identify untransformed data in the shared memory that matches the requested data, loading, by the processor, the requested data from the storage system via the I/O load operation.

3. The method of claim 2 , further comprising:
responsive to failing to identify untransformed data in the shared memory that matches the requested data and prior to loading the requested data from the storage system via the I/O load operation, searching, by the processor, transformed data in a global memory to identify data that matches the requested data;
responsive to identifying transformed data in the global memory that matches the requested data, using, by the processor, the transformed data in the global memory to fulfill the request;
responsive to failing to identify transformed data in the global memory that matches the requested data, searching, by the processor, untransformed data in the global memory to identify data that matches the requested data;
responsive to identifying untransformed data in the global memory that matches the requested data, using, by the processor, the untransformed data in the global memory to fulfill the request; and
responsive to failing to identify untransformed data in the global memory that matches the requested data, loading, by the processor, the requested data from the storage system via the I/O load operation.

4. The method of claim 1 , further comprising:
responsive to a portion of data in the local memory being utilized by more than one application, marking, by the processor, the portion of data such that the portion of data is persisted within the local memory for a longer time than unmarked portions of data.

5. The method of claim 1 , further comprising:
responsive to a portion of data in the local memory being utilized by multiple processing systems within a same block each running an application in the data processing system above a predetermined frequency threshold, moving, by the processor, the portion of data from the local memory to a shared memory, the shared memory being accessible by the multiple processing systems; and
redirecting, by the processor, accesses to the portion of data to the shared memory.

6. The method of claim 1 , further comprising:
responsive to identifying a data stream to be utilized by multiple processing systems within a same block each running an application in the data processing system, loading, by the processor, the data stream into a shared memory, the shared memory being accessible by the multiple processing systems; and
redirecting, by the processor, accesses to the data stream to the shared memory.

7. The method of claim 4 , wherein identifying the data stream to be utilized by the multiple processing systems comprises:
performing, by the processor, a static code analysis of each application running on each of the multiple processing systems to identify one or more data streams to be utilized;
responsive to two or more applications utilizing a same data stream, identifying, by the processor, the data stream as the data stream to be utilized by the multiple processing systems.

8. The method of claim 1 , further comprising:
responsive to a portion of data in the local memory or a shared memory being utilized by multiple processing systems within a different blocks each running an application in the data processing system above a predetermined frequency threshold, moving, by the processor, the portion of data from the local memory or the shared memory to a global memory, the global memory, being accessible all the processing systems; and
redirecting, by the processor, accesses to the portion of data to the global memory.

Assignees

• IBM

Inventors

• Dean Daniel J
• Karve Alexei A
• Lei Hui
• Sailer Anca

Classifications

• G06F3/0656

  Data buffering arrangements · CPC title

• G06F3/06

  Digital input from, or digital output to, record carriers {, e.g. RAID, emulated record carriers or networked record carriers} · CPC title

• G06F3/061

  Improving I/O performance · CPC title

• G06F3/067 Primary

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F16/24

  Querying · CPC title