Data pruning based on metadata

Patent metadata

Field	Value
Publication number	US-11726959-B2
Application number	US-201916399448-A
Country	US
Kind code	B2
Filing date	Apr 30, 2019
Priority date	Jul 14, 2016
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

A system, apparatus, and method for processing queries wherein the query includes a request to access or delete data and accessing metadata associated with the set of data, the metadata defining data characteristics of the set of data and identifying at least sets of data that need or not need to be accessed or deleted based on the metadata without accessing the actual data in the set of data; also methods to optimize processing of some operations based on the collected metadata on data.

First claim

The invention claimed is:

1. A system comprising:
a metadata store to store metadata, wherein the metadata include a summary of data in remote data storage systems as well as data from local cache; and
a processor, operatively coupled with the metadata store, to:
determine whether each of a plurality of files that comprises database data satisfies at least one of a plurality of predicates by reading the metadata stored in the metadata store that is separate from the database data, wherein the metadata includes range data for content corresponding to multiple columns of the database data, the range data including a plurality of different types of data types of the range data, and the range data further includes multiple ones of the plurality of different types of data types of the range data for at least one of the plurality of files; and
prune inapplicable files from the plurality of files that do not satisfy at least one of the plurality of predicates to create a reduced set of files.

2. The system of claim 1 , wherein the processor further to read the reduced set of files to execute a query received by the processor.

3. The system of claim 1 , wherein the processor to remove, from the query, the one or more predicates that do not fully match any file in the plurality of files to create a modified query.

4. The system of claim 1 , wherein the processor to store the database data in the plurality of files, wherein each file of the plurality of files comprises a discrete selection of the database data.

5. The system of claim 2 , wherein to prune the inapplicable files, the processor to:
identify the inapplicable files without accessing the inapplicable files.

6. The system of claim 2 , wherein to read the metadata, the processor to:
determine, based on the metadata, a range of values of database data stored in a certain file; and
determine whether any data within the range of values of database data stored in the certain file satisfies at least one of the plurality of predicates.

7. The system of claim 2 , wherein the processor further to execute the query.

8. A method comprising:
determining, by a processor, whether each of a plurality of files that comprises database data satisfies at least one of a plurality of predicates by reading metadata stored in a metadata store that is separate from the database data, wherein the metadata include a summary of data in remote data storage systems as well as data from local cache and include range data for content corresponding to multiple columns of the database data, the range data including a plurality of different types of data types of the range data, and the range data further includes multiple ones of the plurality of different types of data types of the range data for at least one of the plurality of files; and
pruning inapplicable files from the plurality of files that do not satisfy at least one of the plurality of predicates to create a reduced set of files.

9. The method of claim 8 , further comprising reading the reduced set of files to execute a query.

10. The method of claim 8 , further comprising removing, from the query, the one or more predicates that do not fully match any file in the plurality of files to create a modified query.

11. The method of claim 8 , further comprising storing the database data in the plurality of files, wherein each file of the plurality of files comprises a discrete selection of the database data.

12. The method of claim 9 , wherein pruning the inapplicable files comprises:
identifying the inapplicable files without accessing the inapplicable files; and
removing from consideration the inapplicable files without the inapplicable files being read during execution of the query.

13. The method of claim 9 , wherein reading the metadata comprises:
determining, based on the metadata, a range of values of database data stored in a certain file; and
determining whether any data within the range of values of database data stored in the certain file satisfies at least one of the plurality of predicates.

14. The method of claim 9 , further comprising executing the query.

15. A non-transitory computer readable storage media, programmable to execute instructions that, when executed by a processor, cause the processor to:
determine whether each of a plurality of files that comprises database data satisfies at least one of a plurality of predicates by reading metadata stored in a metadata store that is separate from the database data, wherein the metadata include a summary of data in remote data storage systems as well as data from local cache and include range data for content corresponding to multiple columns of the database data, the range data including a plurality of different types of data types of the range data, and the range data further includes multiple ones of the plurality of different types of data types of the range data for at least one of the plurality of files; and
prune, by the processor, inapplicable files from the plurality of files that do not satisfy at least one of the plurality of predicates to create a reduced set of files.

16. The non-transitory computer readable storage media of claim 15 , the instructions further to cause the processor to read the reduced set of files to execute a query received by the processor.

17. The non-transitory computer readable storage media of claim 15 , the instructions further to cause the processor to remove, from the query, the one or more predicates that do not fully match any file in the plurality of files to create a modified query.

18. The non-transitory computer readable storage media of claim 15 , the instructions further to cause the processor to store the database data in the plurality of files, wherein each file of the plurality of files comprises a discrete selection of the database data.

19. The non-transitory computer readable storage media of claim 16 , wherein to prune the inapplicable files, the processor to:
identify the inapplicable files without accessing the inapplicable files; and
remove from consideration the inapplicable files without the inapplicable files being read during execution of the query.

20. The non-transitory computer readable storage media of claim 16 , wherein to read the metadata, the processor to:
determine, based on the metadata, a range of values of database data stored in a certain file; and
determine whether any data within the range of values of database data stored in the certain file satisfies at least one of the plurality of predicates.

21. The non-transitory computer readable storage media of claim 16 , wherein the processor further to execute the query.

Assignees

• Snowflake Inc

Inventors

• Zukowski Marcin
• Dageville Benoit
• Cruanes Thierry
• Motivala Ashish

Classifications

• G06F16/162 Primary

  Delete operations (erasing in storage systems G06F3/0652) · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/24544

  Join order optimisation · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title