Data pruning based on metadata

Patent metadata

Field	Value
Publication number	US-2019258607-A1
Application number	US-201916399448-A
Country	US
Kind code	A1
Filing date	Apr 30, 2019
Priority date	Jul 14, 2016
Publication date	Aug 22, 2019
Grant date	—

Abstract

A system, apparatus, and method for processing queries wherein the query includes a request to access or delete data and accessing metadata associated with the set of data, the metadata defining data characteristics of the set of data and identifying at least sets of data that need or not need to be accessed or deleted based on the metadata without accessing the actual data in the set of data; also methods to optimize processing of some operations based on the collected metadata on data.

First claim

1 . A system comprising:
means for receiving a query comprising a plurality of predicates; means for identifying one or more applicable files comprising database data satisfying at least one of the plurality of predicates; means for pruning inapplicable files comprising database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files; and means for reading the reduced set of files to execute the query; wherein the means for identifying the one or more applicable files comprises means for reading metadata stored in a metadata store that is separate from the database data.

2 . The system of claim 1 , further comprising means for identifying based on the metadata store one or more predicates of the query that do not fully match any file comprising database data.

3 . The system of claim 2 , further comprising means for removing from the query the one or more predicates that do not fully match any file comprising database data to create a modified query.

4 . The system of claim 1 , further comprising means for storing the database data in a plurality of files, wherein each file of the plurality of files comprises a discrete selection of the database data.

5 . The system of claim 4 , further comprising:
means for generating a metadata file comprising information about a discrete selection of database data stored in a file; and means for storing the metadata file in the metadata store separate from the discrete selection of the database data.

6 . The system of claim 1 , wherein the means for pruning the inapplicable files comprises:
means for identifying the inapplicable files without accessing the inapplicable files; and means for removing from consideration the inapplicable files such that the inapplicable files are not read during execution of the query.

7 . The system of claim 6 , wherein the means for identifying the inapplicable files comprises one or more of:
means for reading metadata pertaining to each of the inapplicable files, the metadata stored in the metadata store; or means for identifying all other files comprising database data not including the applicable files identified by the means for identifying the one or more applicable files.

8 . The system of claim 1 , wherein the means for reading the metadata comprises:
means for determining, based on the metadata, a range of values of database data stored in a certain file; and means for determining whether any data within the range of values of database data stored in the certain file satisfies at least one of the plurality of predicates.

9 . The system of claim 8 , wherein the means for identifying the one or more applicable files comprises means for identifying the certain file as being an applicable file in response to a determination that some data within the range of values stored in the certain file satisfies at least one of the plurality of predicates.

10 . The system of claim 1 , further comprising means for identifying at least one applicable column within each of the one or more applicable files that comprises database data satisfying at least one of the plurality of predicates.

11 . A method comprising:
receiving a query comprising a plurality of predicates; identifying one or more applicable files comprising database data satisfying at least one of the plurality of predicates; pruning inapplicable files comprising database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files; and reading the reduced set of files to execute the query; wherein the identifying the one or more applicable files comprises reading metadata stored in a metadata store that is separate from the database data.

12 . The method of claim 11 , further comprising:
identifying, based on the metadata, one or more predicates of the query that do not fully match any file comprising database data; and removing from the query the one or more predicates that do not fully match any file comprising database data to create a modified query.

13 . The method of claim 11 , further comprising storing the database data in a plurality of files, wherein each file of the plurality of files comprises a discrete selection of the database data.

14 . The method of claim 11 , wherein pruning the inapplicable files comprises:
identifying the inapplicable files without accessing the inapplicable files; and removing from consideration the inapplicable files such that the inapplicable files are not read during execution of the query.

15 . The method of claim 14 , wherein reading the metadata comprises:
determining, based on the metadata, a range of values of database data stored in a certain file; and determining whether any data within the range of values of database data stored in the certain file satisfies at least one of the plurality of predicates.

16 . A processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
receiving a query comprising a plurality of predicates; identifying one or more applicable files comprising database data satisfying at least one of the plurality of predicates; pruning inapplicable files comprising database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files; and reading the reduced set of files to execute the query; wherein the identifying the one or more applicable files comprises reading metadata stored in a metadata store that is separate from the database data.

17 . The processor of claim 16 , the instructions further comprising:
identifying, based on the metadata, one or more predicates of the query that do not fully match any file comprising database data; and removing from the query the one or more predicates that do not fully match any file comprising database data to create a modified query.

18 . The processor of claim 16 , the instructions further comprising storing the database data in a plurality of files, wherein each file of the plurality of files comprises a discrete selection of the database data.

19 . The processor of claim 16 , wherein reading the metadata comprises:
determining, based on the metadata, a range of values of database data stored in a certain file; and determining whether any data within the range of values of database data stored in the certain file satisfies at least one of the plurality of predicates.

20 . The processor of claim 16 , wherein pruning the inapplicable files comprises:
identifying the inapplicable files without accessing the inapplicable files; and removing from consideration the inapplicable files such that the inapplicable files are not read during execution of the query.

Assignees

• Snowflake Inc

Inventors

• Zukowski Marcin
• Dageville Benoit
• Cruanes Thierry
• Motivala Ashish

Classifications

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/162 Primary

  Delete operations (erasing in storage systems G06F3/0652) · CPC title

• G06F16/24544

  Join order optimisation · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title