Accessing tables with heterogeneous partitions

Patent metadata

Field	Value
Publication number	US-10642741-B2
Application number	US-201715425808-A
Country	US
Kind code	B2
Filing date	Feb 6, 2017
Priority date	Feb 6, 2017
Publication date	May 5, 2020
Grant date	May 5, 2020

Abstract

A computer-implemented method according to one embodiment includes receiving a request for data, locating the data at one or more partitions of a heterogeneously partitioned table, determining an access method associated with each of the one or more partitions, and requesting the data from the one or more partitions, utilizing the access method associated with each of the one or more partitions.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a single request to read data;
locating the data requested by the single request at both a first partition and a second partition of a heterogeneously partitioned table, where the first partition represents a first storage location that utilizes a column organized storage format to store data, and the second partition represents a second storage location that utilizes a row organized storage format to store data, and the data is currently being transferred from one partition to another partition within the heterogeneously partitioned table;
determining a first access method associated with the first partition, based on a description of the first partition including a storage format used by the first partition, a location of the first partition, a type of hardware of the first partition, and a data type used by the first partition, where the description of the first partition is stored as first metadata within a predetermined location;
determining a second access method associated with the second partition, based on a description of the second partition including a storage format used by the second partition, a location of the second partition, a type of hardware of the second partition, and a data type used by the second partition, where the description of the second partition is stored as second metadata within the predetermined location, where the second access method is different from the first access method; and
requesting the data from the first partition and the second partition, utilizing the first access method and the second access method, including:
creating, in parallel, a first thread for the first partition and a second thread for the second partition,
instantiating a first sub-operator on the first thread for the first partition, where the first sub-operator requests a first portion of the data on the first partition utilizing the first access method, and
instantiating a second sub-operator on the second thread for the second partition, where the second sub-operator requests a second portion of the data on the second partition utilizing the second access method.

2. The computer-implemented method of claim 1 , wherein the data is stored in a combination of storage mediums and object-stores.

3. The computer-implemented method of claim 1 , wherein the first storage location is different from the second storage location.

4. The computer-implemented method of claim 1 , wherein the first storage location is cloud-based, and the second storage location is on-premises enterprise-based.

5. The computer-implemented method of claim 1 , further comprising determining when data overlap occurs, and inserting one or more filters to avoid duplicate counting.

6. The computer-implemented method of claim 1 , further comprising identifying and eliminating one or more additional partitions of the heterogeneously partitioned table that are not needed in response to the single request.

7. The computer-implemented method of claim 1 , wherein:
during the transfer, a location of the data being transferred is tracked by identifying one or more timestamps for the data being transferred, and using the one or more timestamps to identify a current location of the data within the heterogeneously partitioned table.

8. The computer-implemented method of claim 1 , further comprising determining when data overlap occurs, and inserting one or more filters to avoid duplicate counting, wherein:
the first storage location is cloud-based, and the second storage location is on-premises enterprise-based.

9. A computer program product for accessing tables with heterogeneous partitions, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:
receiving a single request to read data, utilizing the processor;
locating the data requested by the single request at both a first partition and a second partition of a heterogeneously partitioned table, utilizing the processor, where the first partition represents a first storage location that utilizes a column organized storage format to store data, and the second partition represents a second storage location that utilizes a row organized storage format to store data, and the data is currently being transferred from one partition to another partition within the heterogeneously partitioned table;
determining, utilizing the processor, a first access method associated with the first partition, based on a description of the first partition including a storage format used by the first partition, a location of the first partition, a type of hardware of the first partition, and a data type used by the first partition, where the description of the first partition is stored as first metadata within a predetermined location;
determining, utilizing the processor, a second access method associated with the second partition, based on a description of the second partition including a storage format used by the second partition, a location of the second partition, a type of hardware of the second partition, and a data type used by the second partition, where the description of the second partition is stored as second metadata within the predetermined location, where the second access method is different from the first access method; and
requesting, utilizing the processor, the data from the first partition and the second partition, utilizing the first access method and the second access method, including:
creating, in parallel, a first thread for the first partition and a second thread for the second partition,
instantiating a first sub-operator on the first thread for the first partition, where the first sub-operator requests a first portion of the data on the first partition utilizing the first access method, and
instantiating a second sub-operator on the second thread for the second partition, where the second sub-operator requests a second portion of the data on the second partition utilizing the second access method.

10. The computer program product of claim 9 , wherein the data is stored in a combination of storage mediums and object-stores.

11. The computer program product of claim 9 , wherein the first storage location is different from the second storage location.

12. The computer program product of claim 9 , wherein the first storage location is cloud-based, and the second storage location is on-premises enterprise-based.

13. A system, comprising:
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
receive a single request to read data;
locate the data requested by the single request at both a first partition and a second partition of a heterogeneously partitioned table, where the first partition represents a first storage location that utilizes a column organized storage format to store data, and the second partition represents a second storage location that utilizes a row organized storage format to store data, and the data is currently being transferred from one partition to another partition within the heterogeneously partitioned table;
determine a first access method associated with the first partition, based on a description of the first partition including a storage format used by the first partition, a location of the first partition, a type of hardware of the first partition, and a data typ

Assignees

• IBM

Inventors

• Floratou Avrilia
• Ozcan Fatma
• Pirahesh Mir H
• Potti Navneet S

Classifications

• G06F2212/282

  Partitioned cache · CPC title

• G06F12/0848 Primary

  Partitioned cache, e.g. separate instruction and operand caches · CPC title

• G06F2212/1041

  Resource optimization · CPC title