Low-latency database system

Patent metadata

Field	Value
Publication number	US-12130810-B2
Application number	US-202218083157-A
Country	US
Kind code	B2
Filing date	Dec 16, 2022
Priority date	Aug 19, 2020
Publication date	Oct 29, 2024
Grant date	Oct 29, 2024

Abstract

A database system comprised of a decoupled compute layer and storage layer is implemented to store, build, and maintain a canonical dataset, a temporary buffer, and an edits dataset. The canonical dataset is a set of batch updated data. The data is appended in chunks to the canonical dataset such that the canonical dataset becomes a historical dataset over time. The buffer is a write ahead log that contains the most recent chunks of data and provides atomicity and durability for the database system. The edits dataset is the set of data that contains edits such as cell mutations, row appends and/or row deletions. The database system enables users to make cell or row-level edits to tables and observe those edits in analytical systems or downstream builds with minimal latency.

First claim

What is claimed is:

1. A system comprising:
one or more non-transitory computer readable storage mediums having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the system to:
receive one or more edits to be made to a first dataset;
append the one or more edits to a second dataset; and
in response to receipt of a query of the first dataset:
collapse the second dataset based on one or more resolution rules;
consolidate the collapsed second dataset with the first dataset; and
execute the query on the consolidated dataset.

2. The system of claim 1 , wherein consolidating the collapsed second dataset with the first dataset includes joining the collapsed second dataset with the first dataset.

3. The system of claim 1 , wherein consolidating the collapsed second dataset with the first dataset includes combining the collapsed second dataset with the first dataset.

4. The system of claim 1 , wherein consolidating the collapsed second dataset with the first dataset includes merging the collapsed second dataset with the first dataset.

5. The system of claim 1 , wherein the one or more resolution rules specify that, for any data item, a most-recent edit for that data item is to be kept in the second dataset.

6. The system of claim 1 , wherein collapsing the second dataset includes determining a most-recent edit associated with each row in the second dataset.

7. The system of claim 1 , wherein collapsing the second dataset includes de-duplicating data of the second dataset, removing one or more rows of the second dataset, removing one or more columns of the second dataset, or sorting data of the second dataset.

8. The system of claim 1 , wherein the one or more processors are configured to execute the program instructions to further cause the system to:
receive one or more additional edits to be made to the first dataset;
temporarily store the one or more additional edits in a buffer; and
in response to receipt of a second query of the first dataset:
join or combine the buffer to the second dataset to form a joined or combined second dataset;
collapse the joined or combined second dataset based on one or more second resolution rules;
consolidate the collapsed joined or combined second dataset with the first dataset to form a second consolidated dataset;
execute the second query on the second consolidated dataset; and
return a result of the second query.

9. The system of claim 1 , wherein the one or more edits to be made to the first dataset are provided by a user via a user interface.

10. The system of claim 1 , wherein the one or more processors are configured to execute the program instructions to further cause the system to:
store the one or more edits in a buffer as buffer data; and
dump at least a portion of the buffer data to the second dataset.

11. The system of claim 1 , wherein the one or more processors are configured to execute the program instructions to further cause the system to:
return a result of the query executed on the consolidated dataset.

12. A computer-implemented method comprising:
receiving one or more edits to be made to a first dataset;
appending the one or more edits to a second dataset; and
in response to receipt of a query of the first dataset:
collapsing the second dataset based on one or more resolution rules;
consolidating the collapsed second dataset with the first dataset; and
executing the query on the consolidated dataset.

13. The computer-implemented method of claim 12 , wherein consolidating the collapsed second dataset with the first dataset includes joining the collapsed second dataset with the first dataset.

14. The computer-implemented method of claim 12 , wherein consolidating the collapsed second dataset with the first dataset includes combining the collapsed second dataset with the first dataset.

15. The computer-implemented method of claim 12 , wherein consolidating the collapsed second dataset with the first dataset includes merging the collapsed second dataset with the first dataset.

16. The computer-implemented method of claim 12 , further comprising:
receiving one or more additional edits to be made to the first dataset;
temporarily storing the one or more additional edits in a buffer; and
in response to receipt of a second query of the first dataset:
joining or combining the buffer to the second dataset to form a joined or combined second dataset;
collapsing the joined or combined second dataset based on one or more second resolution rules;
consolidating the collapsed joined or combined second dataset with the first dataset to form a second consolidated dataset;
executing the second query on the second consolidated dataset; and
returning a result of the second query.

17. Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
receiving one or more edits to be made to a first dataset;
appending the one or more edits to a second dataset; and
in response to receipt of a query of the first dataset:
collapsing the second dataset based on one or more resolution rules;
consolidating the collapsed second dataset with the first dataset; and
executing the query on the consolidated dataset.

18. The non-transitory computer-readable media of claim 17 , wherein the one or more resolution rules specify that, for any data item, a most-recent edit for that data item is to be kept in the second dataset.

19. The non-transitory computer-readable media of claim 17 , wherein collapsing the second dataset includes determining a most-recent edit associated with each row in the second dataset.

20. The non-transitory computer-readable media of claim 17 , wherein collapsing the second dataset includes de-duplicating data of the second dataset, removing one or more rows of the second dataset, removing one or more columns of the second dataset, or sorting data of the second dataset.

Assignees

• Palantir Technologies Inc

Inventors

• Elliot Mark
• Ellis Joseph
• Ramsharan Rahij
• Sills Matthew
• Manning Lawrence

Classifications

• G06F16/2456

  Join operations · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/24564

  Applying rules; Deductive queries · CPC title

• G06F16/1805

  Append-only file systems, e.g. using logs or journals to store data · CPC title

• G06F16/248

  Presentation of query results · CPC title