External data access with split index

Patent metadata

Field	Value
Publication number	US-11030179-B2
Application number	US-201715650827-A
Country	US
Kind code	B2
Filing date	Jul 14, 2017
Priority date	Jan 31, 2014
Publication date	Jun 8, 2021
Grant date	Jun 8, 2021

Abstract

A split-index can be employed for access to external data. The index can be created on a primary data storage system for data stored externally on a secondary data storage system. After creation, the index can be utilized to expedite at least query execution over the externally stored data. The index can be updated upon detection of changes to data. Further, even when the index is not completely up to date, the index can be exploited for query execution. Furthermore, hybrid execution is enabled with the index and without the index.

First claim

What is claimed is:

1. A method, comprising:
employing at least one processor configured to execute computer-executable instructions stored in memory to perform the following acts:
creating an index in a primary data storage system describing data stored externally of the primary data storage system in a secondary data storage system, the secondary data storage system being a non-relational distributed file system;
during a period of time during which the index is available for execution with queries of the data within the non-relational distributed file system:
initiating execution of at least a portion of a query with the index over a first portion of the data described by a first portion of the index determined to be current and without the index over a second portion of the data described by a second portion of the index determined to be out of date by initiating a scan of the second portion of the data within the non-relational distributed file system to obtain a query response; and

performing an incremental update to the index for the second portion of the index describing the second portion of the data for which the scan was initiated.

2. The method of claim 1 , further comprising acquiring data from the secondary data storage system for at least the portion of the query based on the index.

3. The method of claim 2 , further comprising bypassing a native data processing mechanism of the secondary data storage system in acquiring data from the secondary data storage system, wherein the native data processing mechanism of the secondary data storage system includes a MapReduce processing mechanism.

4. The method of claim 1 , further comprising determining whether to execute the query using the index at the primary data storage system based on a selectivity of the query, wherein the selectivity is a probability that the query satisfies a filter expression.

5. The method of claim 1 , wherein performing the incremental update includes updating the index based on one or more changes to the data stored externally in the secondary data storage system.

6. The method of claim 1 , further comprising determining one or more updates to the index based on one or more changes to the data stored externally in the secondary data storage system concurrently with execution of the query at the primary data storage system.

7. The method of claim 6 , wherein performing the incremental update includes committing the one or more updates to the index upon a pause in processing activity at the primary data storage system.

8. The method of claim 1 , wherein the primary data storage system is a relational data warehouse system.

9. A system, comprising:
a processor coupled to a memory, the processor configured to execute computer-executable instruction including the following computer-executable components stored in the memory:
a first component configured to create an index in a primary data storage system describing data stored externally of the primary data storage system in a secondary data storage system, the secondary data storage system being a non-relational distributed file system; and
a second component configured to, during a period of time during which the index is available for execution with queries of the data within the non-relational distributed file system:
initiate execution of at least a portion of a query with the index over a first portion of the data described by a first portion of the index determined to be current and without the index over a second portion of the data described by a second portion of the index determined to be out of date by initiating a scan of the second portion of the data within the non-relational distributed file system to obtain a query response;

the processor being further configured to execute the computer-executable instructions to perform an incremental update to the index for the second portion of the index describing the second portion of the data for which the scan was initiated.

10. The system of claim 9 , wherein the primary data storage system is a relational data warehouse system.

11. The system of claim 9 , wherein:
the second portion of the data has been modified since the index was created.

12. The system of claim 9 , wherein:
a signal indicates that the index is to be utilized in the execution of the query.

13. The system of claim 9 , wherein the second component is configured to perform the incremental update to the index in view of changes to the data.

14. A computer-readable storage device having instructions stored thereon that enable at least one processor to perform a method upon execution of the instructions, the method comprising:
creating an index locally in a relational data warehouse system describing data stored externally of the relational data warehouse system in a non-relational distributed file system;
during a period of time during which the index is available for execution with queries of the data within the non-relational distributed file system:
initiating execution of at least a portion of a query with the index over a first portion of the data described by a first portion of the index determined to be current and without the index over a second portion of the data described by a second portion of the index determined to be out of date by initiating a scan of the second portion of the data within the non-relational distributed file system to obtain a query response; and

performing an incremental update to the index for the second portion of the index describing the second portion of the data for which the scan was initiated.

15. The computer-readable storage device of claim 14 , wherein the method further comprises acquiring at least the first portion of the data from the distributed file system for the portion of the query based on the index.

16. The computer-readable storage device of claim 15 , wherein the method further comprises acquiring the second portion of the data from the distributed file system for the portion of the query not based on the index.

17. The computer-readable storage device of claim 14 , wherein the index incrementally updated in response to one or more changes to the data.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Teletia Nikhil
• Gankidi Vinitha Reddy
• Halverson Alan D
• Patel Jignesh M

Classifications

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F16/41

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/245

  Query processing · CPC title

• G06F16/2272 Primary

  Management thereof · CPC title