Serialization of data in a concurrent transaction processing distributed database

Patent metadata

Field	Value
Publication number	US-2023169068-A1
Application number	US-202117538818-A
Country	US
Kind code	A1
Filing date	Nov 30, 2021
Priority date	Nov 30, 2021
Publication date	Jun 1, 2023
Grant date	—

Abstract

A distributed database system can implement a column-based database system and a row-based database system for processing data. The row-based database system can store data organized into key value pairs, and data to be processed by the row-based database system is converted to a key-value format compressing keys that correspond to values. The distributed database system can perform serialization and compression in converting the data to the key-value format for efficient data storage performance. The distributed database system can unpack portions of the converted serialized compressed data in response to queries that process a portion of serialized compressed data without unpacking the entire converted dataset.

First claim

What is claimed is:

1 . A method comprising:
identifying, in a hybrid database system, column-organized data that comprises primary key data and non-primary key data, the hybrid database system comprising a columnar database system that processes data in a columnar format and a key-value database system that processes key-value data in a key-value format; serializing, by at least one or more hardware processor, the non-primary key data of the column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using one or more bitmaps; storing the compressed serialized value data in a key-value data store of the key-value database system; receiving a query by the columnar database system; deserializing a portion of the compressed serialized value data that corresponds to the query; and processing the query using the columnar database system.

2 . The method of claim 1 , wherein the one or more bitmaps are applied to generate bitmap compressed data.

3 . The method of claim 2 , wherein the compressed serialized value data is generated by further compressing the bitmap compressed data using an additional compression scheme.

4 . The method of claim 3 , wherein the compressed serialized value data comprises metadata that indicates whether the additional compression scheme is applied to the compressed serialized value data.

5 . The method of claim 1 , wherein the key-value database system is an online transactional processing (OLTP) database system and the columnar database system is an online analytical processing (OLAP) database system.

6 . The method of claim 1 , wherein the portion of the compressed serialized value data that corresponds to the query is deserialized without deserializing remaining data in the compressed serialized value data that does not correspond to the query.

7 . The method of claim 1 , wherein the portion of data of the compressed serialized value data is deserialized using the one or more bitmaps.

8 . The method of claim 7 , wherein deserializing the portion of data using the one or more bitmaps decompress the portion of data.

9 . A system comprising:
one or more processors of a machine; and at least one memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising: identifying, in a hybrid database system, column-organized data that comprises primary key data and non-primary key data, the hybrid database system comprising a columnar database system that processes data in a columnar format and a key-value database system that processes key-value data in a key-value format; serializing, by at least one or more hardware processor, the non-primary key data of the column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using one or more bitmaps; storing the compressed serialized value data in a key-value data store of the key-value database system; receiving a query by the columnar database system; deserializing a portion of the compressed serialized value data that corresponds to the query; and processing the query using the columnar database system.

10 . The system of claim 9 , wherein the one or more bitmaps are applied to generate bitmap compressed data.

11 . The system of claim 10 , wherein the compressed serialized value data is generated by further compressing the bitmap compressed data using an additional compression scheme.

12 . The system of claim 11 , wherein the compressed serialized value data comprises metadata that indicates whether the additional compression scheme is applied to the compressed serialized value data.

13 . The system of claim 9 , wherein the key-value database system comprises an online transactional processing (OLTP) database system and the columnar database system comprises an online analytical processing (OLAP) database system.

14 . The system of claim 9 , wherein the portion of the compressed serialized value data that corresponds to the query is deserialized without deserializing remaining data in the compressed serialized value data that does not correspond to the compressed serialized value data.

15 . The system of claim 9 , wherein the portion of data of the compressed serialized value data is deserialized using the one or more bitmaps.

16 . The system of claim 15 , wherein deserializing the portion of data using the one or more bitmaps decompress the portion of data.

17 . A machine storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
identifying, in a hybrid database system, column-organized data that comprises primary key data and non-primary key data, the hybrid database system comprising a columnar database system that processes data in a columnar format and a key-value database system that processes key-value data in a key-value format; serializing, by at least one or more hardware processor, the non-primary key data of the column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using one or more bitmaps; storing the compressed serialized value data in a key-value data store of the key-value database system; receiving a query by the columnar database system; deserializing a portion of the compressed serialized value data that corresponds to the query; and processing the query using the columnar database system.

18 . The machine storage medium of claim 17 , wherein the one or more bitmaps are applied to generate bitmap compressed data.

19 . The machine storage medium of claim 18 , wherein the compressed serialized value data is generated by further compressing the bitmap compressed data using an additional compression scheme.

20 . The machine storage medium of claim 19 , wherein the compressed serialized value data comprises metadata that indicates whether the additional compression scheme is applied to the compressed serialized value data.

Assignees

• Snowflake Inc

Inventors

• Yagoub Khaled
• Zhu Wumengjian
• Dageville Benoit
• Waddington William

Classifications

• G06F16/2379 Primary

  Updates performed during online database operations; commit processing · CPC title

• G06F11/1458

  Management of the backup or restore process · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F16/283

  Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP · CPC title

• G06F16/2237 Primary

  Vectors, bitmaps or matrices · CPC title