Optimizing data retrieval operation in big-data processing systems

Patent metadata

Field	Value
Publication number	US-10614092-B2
Application number	US-201715413706-A
Country	US
Kind code	B2
Filing date	Jan 24, 2017
Priority date	Jan 24, 2017
Publication date	Apr 7, 2020
Grant date	Apr 7, 2020

Abstract

A mechanism is provided for implementing operation optimization. Responsive to a request to load data via a input/output (I/O) load operation, prior to loading the data, transformed data in a local memory is searched for a match to the requested data. Responsive to identifying transformed that matches the requested data, the transformed data in the local memory is used to fulfill the request. Responsive to failing to identify transformed data in the local memory that matches the requested data, untransformed data in the local memory is searched for a match to the requested data. Responsive to identifying untransformed data that matches the requested data, the untransformed data in the local memory is used to fulfill the request. Responsive to failing to identify untransformed data in the local memory that matches the requested data, the requested data may be loaded from the storage system via the I/O load operation.

First claim

What is claimed is:

1. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program for implementing data retrieval operation optimization stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
responsive to a request from an application to load data from a storage system via a input/output (I/O) load operation, prior to loading the data from the storage system, search transformed data in a local memory to identify data that matches the requested data;
responsive to identifying transformed data in the local memory that matches the requested data, use the transformed data in the local memory to fulfill the request;
responsive to failing to identify transformed data in the local memory that matches the requested data, search untransformed data in the local memory to identify data that matches the requested data;
responsive to identifying untransformed data in the local memory that matches the requested data, use the untransformed data in the local memory to fulfill the request; and
responsive to failing to identify untransformed data in the local memory that matches the requested data, load the requested data from the storage system via the I/O load operation, wherein using the transformed data in the local memory to fulfill the request or, if the transformed data in the local memory that matches the requested data fails to exist, using the untransformed data in the local memory to fulfill the request, prior to loading the data from the storage system, optimizes the data retrieval operation.

2. The computer program product of claim 1 , wherein the computer readable program further causes the computing device to:
responsive to failing to identify untransformed data in the local memory that matches the requested data and prior to loading the requested data from the storage system via the I/O load operation, search transformed data in a shared memory to identify data that matches the requested data;
responsive to identifying transformed data in the shared memory that matches the requested data, use the transformed data in the shared memory to fulfill the request;
responsive to failing to identify transformed data in the shared memory that matches the requested data, search untransformed data in the shared memory to identify data that matches the requested data;
responsive to identifying untransformed data in the shared memory that matches the requested data, use the untransformed data in the shared memory to fulfill the request; and
responsive to failing to identify untransformed data in the shared memory that matches the requested data, load the requested data from the storage system via the I/O load operation.

3. The computer program product of claim 2 , wherein the computer readable program further causes the computing device to:
responsive to failing to identify untransformed data in the shared memory that matches the requested data and prior to loading the requested data from the storage system via the I/O load operation, search transformed data in a global memory to identify data that matches the requested data;
responsive to identifying transformed data in the global memory that matches the requested data, use the transformed data in the global memory to fulfill the request;
responsive to failing to identify transformed data in the global memory that matches the requested data, search untransformed data in the global memory to identify data that matches the requested data;
responsive to identifying untransformed data in the global memory that matches the requested data, use the untransformed data in the global memory to fulfill the request; and
responsive to failing to identify untransformed data in the global memory that matches the requested data, load the requested data from the storage system via the I/O load operation.

4. The computer program product of claim 1 , wherein the computer readable program further causes the computing device to:
responsive to a portion of data in the local memory being utilized by multiple processing systems within a same block each running an application in the data processing system above a predetermined frequency threshold, move the portion of data from the local memory to a shared memory, the shared memory being accessible by the multiple processing systems; and
redirect accesses to the portion of data to the shared memory.

5. The computer program product of claim 1 , wherein the computer readable program further causes the computing device to:
responsive to identifying a data stream to be utilized by multiple processing systems within a same block each running an application in the data processing system, load the data stream into a shared memory, the shared memory being accessible by the multiple processing systems; and
redirect accesses to the data stream to the shared memory.

6. The computer program product of claim 1 , wherein the computer readable program further causes the computing device to:
responsive to a portion of data in the local memory or a shared memory being utilized by multiple processing systems within a different blocks each running an application in the data processing system above a predetermined frequency threshold, move the portion of data from the local memory or the shared memory to a global memory, the global memory being accessible all the processing systems; and
redirect accesses to the portion of data to the global memory.

7. The computer program product of claim 1 , wherein the computer readable program further causes the computing device to:
responsive to a portion of data in the local memory being utilized by more than one application, mark the portion of data such that the portion of data is persisted within the local memory for a longer time than unmarked portions of data.

8. The computer program product of claim 7 , wherein the computer readable program to identify the data stream to be utilized by the multiple processing systems further causes the computing device to:
perform a static code analysis of each application running on each of the multiple processing systems to identify one or more data streams to be utilized; and
responsive to two or more applications utilizing a same data stream, identify the data stream as the data stream to be utilized by the multiple processing systems.

9. An apparatus for implementing operation optimization comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to:
responsive to a request from an application to load data from a storage system via a input/output (I/O) load operation, prior to loading the data from the storage system, search transformed data in a local memory to identify data that matches the requested data;
responsive to identifying transformed data in the local memory that matches the requested data, use the transformed data in the local memory to fulfill the request;
responsive to failing to identify transformed data in the local memory that matches the requested data, search untransformed data in the local memory to identify data that matches the requested data;
responsive to identifying untransformed data in the local memory that matches the requested data, use the untransformed data in the local memory to fulfill the request; and
responsive to failing to identify untransformed data in the local memory that matches the requested data, load the requested data from the storage system via the I/O load operation, wherein using the transformed data in the local memory to fulfill the request or, if the transformed data in the local memory that m

Assignees

• IBM

Inventors

• Dean Daniel J
• Karve Alexei A
• Lei Hui
• Sailer Anca

Classifications

• G06F3/0656

  Data buffering arrangements · CPC title

• G06F3/06

  Digital input from, or digital output to, record carriers {, e.g. RAID, emulated record carriers or networked record carriers} · CPC title

• G06F3/061

  Improving I/O performance · CPC title

• G06F3/067 Primary

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title