Schema evolution for key columnar data into row-organized sequences

Patent metadata

Field	Value
Publication number	US-12135697-B2
Application number	US-202318326929-A
Country	US
Kind code	B2
Filing date	May 31, 2023
Priority date	Mar 25, 2022
Publication date	Nov 5, 2024
Grant date	Nov 5, 2024

Abstract

The subject technology generates, by a compute service manager, a schema hash value for a new schema version associated with a new schema version value, the schema hash value based on determining a sum of hash values of a set of attributes of value columns, the set of attributes comprises a column identifier, and a logical type of a column. The subject technology stores a mapping of the schema hash value to the new schema version value for a table in a metadata database. The subject technology stores a new schema entry based on the schema hash value, the new schema version value, and a new column for the table in the metadata database, the metadata database storing multiple entries for different schema versions, each entry including a particular schema hash value for mapping to a corresponding schema version from the different schema versions.

First claim

What is claimed is:

1. A system comprising:
at least one hardware processor; and
a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
generating, by a compute service manager, a schema hash value for a new schema version associated with a new schema version value, the schema hash value based on determining a sum of hash values of a set of attributes of value columns, the set of attributes comprises a column identifier, and a logical type of a column;
storing a mapping of the schema hash value to the new schema version value for a table in a metadata database;
storing a new schema entry based on the schema hash value, the new schema version value, and a new column for the table in the metadata database, the metadata database storing multiple entries for different schema versions, each entry including a particular schema hash value for mapping to a corresponding schema version from the different schema versions;
receiving a first statement to perform a read operation of a particular table, the read operation including a predicate to evaluate on the particular table;
sending a second statement to an execution node for executing the read operation of the particular table;
receiving information corresponding to a result of the read operation, the information including a particular schema version and a set of values;
determining that the particular schema version does not match a current schema version and is not in a cache;
performing a lookup of the particular schema version from the metadata database; and
deserializing a key value pair associated with information corresponding to the result of the read operation.

2. The system of claim 1 , wherein the operations further comprise:
prior to generating the schema hash value, receiving, by the compute service manager, a statement to perform an operation to add a new column into a table.

3. The system of claim 1 , wherein the operations further comprise:
performing, by the compute service manager, an operation to add the new column to the table.

4. The system of claim 1 , wherein the set of attributes further comprises an index of the column, the index of the column corresponding to a column position.

5. The system of claim 1 , wherein the operations further comprise:
receiving a first statement to perform an operation to insert a new row into a particular table, the first statement including a set of values to insert;
determining a schema version value corresponding to a current schema version associated with the particular table;
generating a schema hash value based on the schema version value corresponding to the current schema version associated with the particular table;
determining a mapping of schema hash values to schema version values stored in the metadata database; and
validating the current schema version corresponding to the schema version value based on the schema hash value and the mapping.

6. The system of claim 5 , wherein the operations further comprise:
sending a second statement to an execution node for executing an operation to insert a new row, the second statement in a second language different than a first language of the first statement; and
performing, by the execution node, the operation to insert the new row.

7. The system of claim 1 , wherein the operations further comprise:
determining a set of schema versions associated with the particular table, the set of schema versions including a current schema version and other previous schema versions; and
performing, by the execution node, the read operation of the particular table.

8. The system of claim 1 , wherein the operations further comprise:
receiving a first statement to perform an operation to update a set of values corresponding to a key;
determining a schema version value corresponding to a current schema version associated with a particular table;
validating the current schema version corresponding to a schema version value based on a schema hash value and a mapping of schema hash values to the schema version value;
sending a second statement to an execution node for executing the operation to update the set of values corresponding to the key; and
performing, by the execution node, the operation to update the set of values corresponding to the key.

9. The system of claim 1 , wherein the metadata database stores a table of schema version values corresponding to different schema versions, and a particular schema hash value corresponding to each schema version value from the schema version values, each particular schema hash value being different than other schema hash values.

10. A method comprising:
generating, by a compute service manager, a schema hash value for a new schema version associated with a new schema version value, the schema hash value based on determining a sum of hash values of a set of attributes of value columns, the set of attributes comprises a column identifier, and a logical type of a column;
storing a mapping of the schema hash value to the new schema version value for a table in a metadata database; and
storing a new schema entry based on the schema hash value, the new schema version value, and a new column for the table in the metadata database, the metadata database storing multiple entries for different schema versions, each entry including a particular schema hash value for mapping to a corresponding schema version from the different schema versions;
receiving a first statement to perform a read operation of a particular table, the read operation including a predicate to evaluate on the particular table;
sending a second statement to an execution node for executing the read operation of the particular table;
receiving information corresponding to a result of the read operation, the information including a particular schema version and a set of values;
determining that the particular schema version does not match a current schema version and is not in a cache;
performing a lookup of the particular schema version from the metadata database; and
deserializing a key value pair associated with information corresponding to the result of the read operation.

11. The method of claim 10 , further comprising:
prior to generating the schema hash value, receiving, by the compute service manager, a statement to perform an operation to add a new column into a table.

12. The method of claim 10 , further comprising:
performing, by the compute service manager, an operation to add the new column to the table.

13. The method of claim 10 , wherein the set of attributes further comprises an index of the column, the index of the column corresponding to a column position.

14. The method of claim 10 , further comprising:
receiving a first statement to perform an operation to insert a new row into a particular table, the first statement including a set of values to insert;
determining a schema version value corresponding to a current schema version associated with the particular table;
generating a schema hash value based on the schema version value corresponding to the current schema version associated with the particular table;
determining a mapping of schema hash values to schema version values stored in the metadata database; and
validating the current schema version corresponding to the schema version value based on the schema hash value and the mapping.

15. The method of claim 14 , further comprising:
sending a second statement to an execution node for executing an operation to insert a new row, the second statement in a second language different than a first l

Assignees

• Snowflake Inc

Inventors

• Dageville Benoit
• Hamza Adrian
• Jiang Lishi
• Waddington William
• Yagoub Khaled
• Zhu Wumengjian

Classifications

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F16/2255

  Hash tables · CPC title

• G06F16/213 Primary

  with details for schema evolution support · CPC title