Catalog query framework on distributed key value store

Patent metadata

Field	Value
Publication number	US-12072840-B2
Application number	US-202318179793-A
Country	US
Kind code	B2
Filing date	Mar 7, 2023
Priority date	Oct 29, 2021
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

Techniques for executing show commands are described herein. A plurality of navigation steps is utilized, each navigation step corresponding to a different layer in a database structure and each navigation step including an operator to fetch items from a metadata database up to respective bounded limits. Dependency information is also fetched for objects of the specified object type in the show command. After a set of objects from the last layer are processed, memory for the navigation steps is flushed and the next set of objects are processed.

First claim

The invention claimed is:

1. A method comprising:
receiving a first show command;
in response to the first show command, initiate fetching a first set of first layer objects in a first layer of a metadata database;
stopping execution of the first show command before all objects in the first set of first layer objects are fetched;
inserting a bookmark indicating where the first show command was stopped in relation to fetching the first set of objects;
receiving a second show command;
in response to the second show command, retrieving the bookmark and fetching remaining objects in the first set of first layer objects in the first layer of the metadata database up to a first bounded limit using a first memory space;
fetching a first set of second layer objects in a second layer of the metadata database corresponding to the first set of first layer objects in the first layer up to a second bounded limit using a second memory space;
fetching dependency information associated with the first set of second layer objects;
flushing the second memory space;
fetching a second set of second layer objects in the second layer of the of the metadata database corresponding to the first set of first layer objects in the first layer up to the second bounded limit using the flushed second memory space;
fetching dependency information associated with the second set of second layer objects;
compiling dependency information associated with the first and second sets of second layer objects; and
generating results for the second show command based on the compiled dependency information.

2. The method of claim 1 , wherein the dependency information associated the first set of second layer objects is fetched in a single transaction with the metadata database using bulk reads.

3. The method of claim 1 , wherein the dependency information associated the second set of second layer objects is fetched in a single transaction with the metadata database using bulk reads.

4. The method of claim 1 , further comprising:
determining that there are additional second layer objects corresponding to the first set of first layer objects; and
in response to determining that there are more second layer objects, flushing the second memory space.

5. The method of claim 1 , further comprising:
collecting metrics associated with each navigation step at a centralized location.

6. The method of claim 5 , wherein the metrics include time spent fetching objects in each navigation step.

7. The method of claim 1 , further comprising:
collecting metrics associated with fetching dependency information.

8. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving a first show command;
in response to the first show command, initiate fetching a first set of first layer objects in a first layer of a metadata database;
stopping execution of the first show command before all objects in the first set of first layer objects are fetched;
inserting a bookmark indicating where the first show command was stopped in relation to fetching the first set of objects;
receiving a second show command;
in response to the second show command, retrieving the bookmark and fetching remaining objects in the first set of first layer objects in the first layer of the metadata database up to a first bounded limit using a first memory space;
fetching a first set of second layer objects in a second layer of the metadata database corresponding to the first set of first layer objects in the first layer up to a second bounded limit using a second memory space;
fetching dependency information associated with the first set of second layer objects;
flushing the second memory space;
fetching a second set of second layer objects in the second layer of the of the metadata database corresponding to the first set of first layer objects in the first layer up to the second bounded limit using the flushed second memory space;
fetching dependency information associated with the second set of second layer objects;
compiling dependency information associated with the first and second sets of second layer objects; and
generating results for the second show command based on the compiled dependency information.

9. The machine-storage medium of claim 8 , wherein the dependency information associated the first set of second layer objects is fetched in a single transaction with the metadata database using bulk reads.

10. The machine-storage medium of claim 8 , wherein the dependency information associated the second set of second layer objects is fetched in a single transaction with the metadata database using bulk reads.

11. The machine-storage medium of claim 8 , further comprising:
determining that there are additional second layer objects corresponding to the first set of first layer objects; and
in response to determining that there are more second layer objects, flushing the second memory space.

12. The machine-storage medium of claim 8 , further comprising:
collecting metrics associated with each navigation step at a centralized location.

13. The machine-storage medium of claim 12 , wherein the metrics include time spent fetching objects in each navigation step.

14. The machine-storage medium of claim 8 , further comprising:
collecting metrics associated with fetching dependency information.

15. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving a first show command;
in response to the first show command, initiate fetching a first set of first layer objects in a first layer of a metadata database;
stopping execution of the first show command before all objects in the first set of first layer objects are fetched;
inserting a bookmark indicating where the first show command was stopped in relation to fetching the first set of objects;
receiving a second show command;
in response to the second show command, retrieving the bookmark and fetching remaining objects in the first set of first layer objects in the first layer of the metadata database up to a first bounded limit using a first memory space;
fetching a first set of second layer objects in a second layer of the metadata database corresponding to the first set of first layer objects in the first layer up to a second bounded limit using a second memory space;
fetching dependency information associated with the first set of second layer objects;
flushing the second memory space;
fetching a second set of second layer objects in the second layer of the of the metadata database corresponding to the first set of first layer objects in the first layer up to the second bounded limit using the flushed second memory space;
fetching dependency information associated with the second set of second layer objects;
compiling dependency information associated with the first and second sets of second layer objects; and
generating results for the second show command based on the compiled dependency information.

16. The system of claim 15 , wherein the dependency information associated the first set of second layer objects is fetched in a single transaction with the metadata database using bulk reads.

17. The system of claim 15 , wherein the dependency information associated the second set of second layer objects is fetched in a single transaction with the metadata database using bulk reads.

Assignees

• Snowflake Inc

Inventors

• Chan Lin
• Chen Tianyi
• Gernhardt Robert Bengt Benedikt
• Mahesh Nithin
• Robinson Eric

Classifications

• G06F16/164

  File meta data generation · CPC title

• G06F16/148 Primary

  File search processing · CPC title

• G06F16/256 Primary

  in federated or virtual databases · CPC title