Columnar Techniques for Big Metadata Management

Patent metadata

Field	Value
Publication number	US-2023185816-A1
Application number	US-202318166056-A
Country	US
Kind code	A1
Filing date	Feb 8, 2023
Priority date	Nov 13, 2020
Publication date	Jun 15, 2023
Grant date	—

Abstract

A method for managing big metadata using columnar techniques includes receiving a query request requesting data blocks from a data table that match query parameters. The data table is associated with system tables that each includes metadata for a corresponding data block of the data table. The method includes generating, based on the query request, a system query to return a subset of rows that correspond to the data blocks that match the query parameters. The method further includes generating, based on the query request and the system query, a final query to return a subset of data blocks from the data table corresponding to the subset of rows. The method also includes determining whether any of the data blocks in the subset of data blocks match the query parameters, and returning the matching data blocks when one or more data blocks match the query parameters.

First claim

What is claimed is:

1 . A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
storing, on memory hardware in communication with the data processing hardware, a data table associated with a system table, the system table comprising a plurality of rows, each respective row of the plurality of rows of the system table comprising:
a plurality of data blocks; and
metadata associated with each data block of the plurality of data blocks of the respective row;
receiving, from a client device, a query requesting return of data blocks from the data table that match query parameters; determining that the query parameters satisfy the respective metadata of one or more rows of the plurality of rows; and returning, to the client device, the one or more rows of the plurality of rows.

2 . The method of claim 1 , wherein the data table comprises a fact table and a dimension table, and wherein the query filters the dimension table.

3 . The method of claim 2 , wherein the operations further comprise generating, based on the query, a fact query that filters the fact table.

4 . The method of claim 1 , wherein a portion of the system table is cached in volatile memory.

5 . The method of claim 4 , wherein the operations further comprise, determining, based on access statistics, the portion of the system table cached in volatile memory.

6 . The method of claim 1 , wherein the data table is stored in a column-major format.

7 . The method of claim 1 , wherein the operations further comprise generating, based on the query, a system query to identify, based on the respective metadata, the one or more rows of the plurality of rows that satisfy the query parameters.

8 . The method of claim 7 , wherein the operations further comprise, generating, based on the query and the system query, a final query to return a subset of data blocks from the data table, each data block of the subset of data blocks corresponding to one of the one or more rows of the plurality of rows that satisfy the query parameters.

9 . The method of claim 8 , wherein generating the final query comprises generating a semi-join of the query and the system query.

10 . The method of claim 7 , wherein the system query comprises a falsifiable expression.

11 . A system comprising:
data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
storing, on the memory hardware, a data table associated with a system table, the system table comprising a plurality of rows, each respective row of the plurality of rows of the system table comprising:
a plurality of data blocks; and
metadata associated with each data block of the plurality of data blocks of the respective row;

receiving, from a client device, a query requesting return of data blocks from the data table that match query parameters;
determining that the query parameters satisfy the respective metadata of one or more rows of the plurality of rows; and
returning, to the client device, the one or more rows of the plurality of rows.

12 . The system of claim 11 , wherein the data table comprises a fact table and a dimension table, and wherein the query filters the dimension table.

13 . The system of claim 12 , wherein the operations further comprise generating, based on the query, a fact query that filters the fact table.

14 . The system of claim 11 , wherein a portion of the system table is cached in volatile memory.

15 . The system of claim 14 , wherein the operations further comprise, determining, based on access statistics, the portion of the system table cached in volatile memory.

16 . The system of claim 11 , wherein the data table is stored in a column-major format.

17 . The system of claim 11 , wherein the operations further comprise generating, based on the query, a system query to identify, based on the respective metadata, the one or more rows of the plurality of rows that satisfy the query parameters.

18 . The system of claim 17 , wherein the operations further comprise, generating, based on the query and the system query, a final query to return a subset of data blocks from the data table, each data block of the subset of data blocks corresponding to one of the one or more rows of the plurality of rows that satisfy the query parameters.

19 . The system of claim 18 , wherein generating the final query comprises generating a semi-join of the query and the system query.

20 . The system of claim 17 , wherein the system query comprises a falsifiable expression.

Assignees

• Google Llc

Inventors

• Edara Pavan
• Pasumansky Mosha

Classifications

• G06F12/0875

  with dedicated cache, e.g. instruction or stack · CPC title

• G06F16/2465 Primary

  Query processing support for facilitating data mining operations in structured databases · CPC title

• G06F2212/45

  Caching of specific data in cache memory · CPC title

• G06F16/2455

  Query execution · CPC title

• G06F16/221 Primary

  Column-oriented storage; Management thereof · CPC title