Low-latency database system

Patent metadata

Field	Value
Publication number	US-12346320-B2
Application number	US-202418891642-A
Country	US
Kind code	B2
Filing date	Sep 20, 2024
Priority date	Aug 19, 2020
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

A computer system can receive one or more edits to be made to a canonical dataset and can temporarily store the one or more edits in a buffer. In response to receipt of a query of the canonical dataset, the computer system can rewrite the query to read from the canonical dataset and the buffer; combine the one or more edits from the buffer with the canonical dataset to form a combined dataset based on resolution policies to avoid conflicts between data; rewrite the query to execute on the combined dataset in lieu of the canonical dataset to optimize query performance; and execute the query on the combined dataset.

First claim

What is claimed is:

1. A system comprising:
one or more non-transitory computer readable storage mediums having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the system to:
receive one or more edits to be made to a canonical dataset;
temporarily store the one or more edits in a buffer; and
in response to receipt of a query of the canonical dataset:
rewrite the query to read from the canonical dataset and the buffer;
combine the one or more edits from the buffer with the canonical dataset to form a combined dataset, wherein combining the one or more edits with the canonical dataset is governed by one or more resolution policies to avoid conflicts between data;
rewrite the query to execute on the combined dataset in lieu of the canonical dataset to optimize query performance; and
execute the query on the combined dataset.

2. The system of claim 1 wherein combining the one or more edits with the canonical dataset comprises joining or merging the one or more edits with the canonical dataset.

3. The system of claim 1 wherein combining the one or more edits from the buffer with the canonical dataset includes collapsing the one or more edits based on the one or more resolution policies.

4. The system of claim 3 wherein collapsing the one or more edits includes de-duplicating data, removing one or more rows of data, removing one or more columns of data, or sorting data.

5. The system of claim 1 wherein the one or more processors are configured to execute the program instructions to cause the system to periodically flush the one or more edits from the buffer.

6. The system of claim 1 wherein the one or more processors are configured to execute the program instructions to cause the system to:
periodically append the one or more edits to an edits dataset; and
combine the one or more edits with the canonical dataset based on combining the edits dataset with the canonical dataset.

7. The system of claim 1 wherein the one or more resolution policies include a default resolution policy to prefer data values from the one or more edits over data values from the canonical dataset when a conflict exists.

8. The system of claim 1 wherein the one or more resolution policies include a time-based resolution policy to prefer more recent data values when a conflict exists between the canonical dataset and the one or more edits.

9. The system of claim 1 wherein the one or more resolution policies include a user-defined function for choosing data values when a conflict exists between the canonical dataset and the one or more edits.

10. A computer-implemented method comprising:
receiving one or more edits to be made to a canonical dataset;
temporarily storing the one or more edits in a buffer; and
in response to receipt of a query of the canonical dataset:
rewriting the query to read from the canonical dataset and the buffer;
combining the one or more edits from the buffer with the canonical dataset to form a combined dataset, wherein combining the one or more edits with the canonical dataset is governed by one or more resolution policies to avoid conflicts between data;
rewriting the query to execute on the combined dataset in lieu of the canonical dataset to optimize query performance; and
executing the query on the combined dataset.

11. The computer-implemented method of claim 10 wherein combining the one or more edits with the canonical dataset comprises joining or merging the one or more edits with the canonical dataset.

12. The computer-implemented method of claim 10 wherein combining the one or more edits from the buffer with the canonical dataset includes collapsing the one or more edits based on the one or more resolution policies.

13. The computer-implemented method of claim 10 further comprising periodically flushing the one or more edits from the buffer.

14. The computer-implemented method of claim 10 further comprising:
periodically appending the one or more edits to an edits dataset; and
combining the one or more edits with the canonical dataset based on combining the edits dataset with the canonical dataset.

15. The computer-implemented method of claim 10 wherein the one or more resolution policies include one or more of a default resolution policy, a time-based resolution policy, or a user-defined resolution policy.

16. Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
receiving one or more edits to be made to a canonical dataset;
temporarily storing the one or more edits in a buffer; and
in response to receipt of a query of the canonical dataset:
rewriting the query to read from the canonical dataset and the buffer;
combining the one or more edits from the buffer with the canonical dataset to form a combined dataset, wherein combining the one or more edits with the canonical dataset is governed by one or more resolution policies to avoid conflicts between data;
rewriting the query to execute on the combined dataset in lieu of the canonical dataset to optimize query performance; and
executing the query on the combined dataset.

17. The non-transitory computer-readable media of claim 16 wherein combining the one or more edits with the canonical dataset comprises joining or merging the one or more edits with the canonical dataset.

18. The non-transitory computer-readable media of claim 16 wherein combining the one or more edits from the buffer with the canonical dataset includes collapsing the one or more edits based on the one or more resolution policies.

19. The non-transitory computer-readable media of claim 16 wherein the computer-executable instructions, when executed, cause the computing system to perform operations comprising:
periodically appending the one or more edits to an edits dataset;
periodically flushing the one or more edits from the buffer; and
combining the one or more edits with the canonical dataset based on combining the edits dataset with the canonical dataset.

20. The non-transitory computer-readable media of claim 16 wherein the one or more resolution policies include one or more of a default resolution policy, a time-based resolution policy, or a user-defined resolution policy.

Assignees

• Palantir Technologies Inc

Inventors

• Elliot Mark
• Ellis Joseph
• Ramsharan Rahij
• Sills Matthew
• Manning Lawrence

Classifications

• G06F16/2456

  Join operations · CPC title

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/24564

  Applying rules; Deductive queries · CPC title

• G06F16/1805

  Append-only file systems, e.g. using logs or journals to store data · CPC title

• G06F16/248

  Presentation of query results · CPC title