Optimizations to read and write transactions for large values in distributed databases

Patent metadata

Field	Value
Publication number	US-12468685-B2
Application number	US-202117555197-A
Country	US
Kind code	B2
Filing date	Dec 17, 2021
Priority date	Dec 17, 2021
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

The subject technology receives a key-value pair. The subject technology determines a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space. The subject technology performs a write operation of a key-pointer pair into a regular space. The subject technology performs a plurality of write operations for multiple key-value parts of the key-value pair into the OLB key space. The subject technology performs conflict checking of the regular space for the key-value pair.

First claim

What is claimed is:

1 . A system comprising:
at least one hardware processor; and a memory storing instructions that cause the at least one hardware processor to perform operations comprising: receiving a statement including a key-value pair, the statement comprising a command to perform a write operation on a linearizable database based on the key-value pair; determining a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space, the OLB key space being provided by a cloud storage platform; performing, by a transaction manager, a first write operation of a key-pointer pair into a regular space, the transaction manager being separate from the cloud storage platform, the cloud storage platform providing the regular space, the key-pointer pair comprising a key, and a stamp, the stamp comprising a transaction identifier assigned to an SQL-level transaction that persists across multiple statement executions and restart attempts within a same transaction, a statement number identifying a SQL statement within the SQL-level transaction, and a restart count, the restart count tracking which statement restart attempt generated the first write operation, the restart count being incremented for each restart of the statement while the stamp maintains a same transaction identifier as the transaction identifier; performing, using the stamp, a plurality of second write operations for multiple key-value parts of the key-value pair into the OLB key space; and performing conflict checking of the regular space for the key-value pair.

2 . The system of claim 1 , wherein the OLB key space and the regular space are different storage spaces in the cloud storage platform, the statement comprises an SQL statement, and the write operation comprising a particular SQL statement to perform an insert or an update to the linearizable database, the transaction manager is included in a compute service manager, the compute service manager executes in a network-based data warehouse system, the compute service manager receiving the SQL statement from a client device, the client device being separate from the compute service manager and the network-based data warehouse system.

3 . The system of claim 1 , wherein the linearizable database comprises a distributed database, the distributed database is stored in the cloud storage platform, the cloud storage platform is a remote system from a compute service manager that has received the statement, the compute service manager executes in a network-based data warehouse system, the compute service manager receives the statement from a client device, the client device being separate from the compute service manager and the network-based data warehouse system, and the statement comprises an SQL statement.

4 . The system of claim 1 , wherein the key-pointer pair further comprises a set of control bits, a value indicating a number of parts of the key-value pair, a pointer, the pointer being based on the stamp, the set of control bits including one bit to indicate whether the key corresponds to the pointer to the OLB key space, a worker identifier, and a write number.

5 . The system of claim 4 , wherein performing the plurality of second write operations for multiple key-value parts into the OLB key space comprises:
performing a particular write operation of first data in the OLB key space, the first data comprising the key, a first pointer, and a first part from the multiple key-value parts of the key-value pair.

6 . The system of claim 5 , wherein the first pointer comprises the stamp and a value indicating a part number from the multiple key-value parts of the key-value pair.

7 . The system of claim 1 , wherein the multiple key-value parts are generated by splitting a large value from the key-value pair into different parts, the large value comprising a particular value greater than the threshold size.

8 . The system of claim 1 , wherein the operations further comprise:
receiving a particular request to perform a read operation on a key; locating a visible version of the key in the regular space; determining that a tombstone bit has not been set of the key; determining that an OLB bit is set for the visible version of the key in the regular space; performing a read operation based on a value of the visible version of the key in the OLB key space; reassembling a large value based on the read operation of the OLB key space; and providing the reassembled large value in response to the particular request.

9 . A method comprising:
receiving a statement including a key-value pair, the statement comprising a command to perform a write operation on a linearizable database based on the key-value pair; determining a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space, the OLB key space being provided by a cloud storage platform; performing, by a transaction manager, a first write operation of a key-pointer pair into a regular space, the transaction manager being separate from the cloud storage platform, the cloud storage platform providing the regular space, the key-pointer pair comprising a key, and a stamp, the stamp comprising a transaction identifier assigned to an SQL-level transaction that persists across multiple statement executions and restart attempts within a same transaction, a statement number identifying a SQL statement within the SQL-level transaction, and a restart count, the restart count tracking which statement restart attempt generated the first write operation, the restart count being incremented for each restart of the statement while the stamp maintains a same transaction identifier as the transaction identifier; performing, using the stamp, a plurality of second write operations for multiple key-value parts of the key-value pair into the OLB key space; and performing conflict checking of the regular space for the key-value pair.

10 . The method of claim 9 , wherein the OLB key space and the regular space are different storage spaces in the cloud storage platform.

11 . The method of claim 9 , wherein the linearizable database comprises a distributed database, the distributed database is stored in the cloud storage platform, and the cloud storage platform is a remote system from a compute service manager that has received the statement.

12 . The method of claim 9 , wherein the key-pointer pair further comprises a set of control bits, a value indicating a number of parts of the key-value pair, a pointer, the pointer being based on the stamp, the set of control bits including one bit to indicate whether the key corresponds to the pointer to the OLB key space, a worker identifier, and a write number.

13 . The method of claim 12 , wherein performing the plurality of second write operations for multiple key-value parts into the OLB key space comprises:
performing a particular write operation of first data in the OLB key space, the first data comprising the key, a first pointer, and a first part from the multiple key-value parts of the key-value pair.

14 . The method of claim 13 , wherein the first pointer comprises the stamp and a value indicating a part number from the multiple key-value parts of the key-value pair.

15 . The method of claim 9 , wherein the multiple key-value parts are generated by splitting a large value from the key-value pair into different parts, the large value comprising a particular value greater than the threshold size.

16 . A non-transitory computer-storage medium comprising instructions that, when executed by o

Assignees

• Snowflake Inc

Inventors

• Luo Chen
• Miller Alexander
• Waddington William
• Xu Mingze

Classifications

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/2379 Primary

  Updates performed during online database operations; commit processing · CPC title

• G06F16/2365 Primary

  Ensuring data consistency and integrity · CPC title