Checkpoints in batch file processing

Patent metadata

Field	Value
Publication number	US-11461325-B2
Application number	US-202117333358-A
Country	US
Kind code	B2
Filing date	May 28, 2021
Priority date	Jun 1, 2020
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

Embodiments of the present disclosure may provide a dynamic query execution model with fault tolerance and failure recovery techniques. Embodiments of the present disclosure may utilize checkpoints to map processed output files to their corresponding input files. Therefore, if an error occurs in processing one or more files, the system may only need to reschedule processing of selected file(s).

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, an assignment of a batch from a query coordinator, the batch including a group of a set of files from a shared file queue;
storing a unique batch ID and information related to the batch in a staging area;
performing a first operator on the batch;
based on performing the first operator, generating a checkpoint signal;
determining a state condition of a second operator as being stateless;
based on the stateless condition of the second operator, performing the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;
determining a state condition of the third operator as being non-stateless;
based on the non-stateless condition of the third operator, clearing data associated with the third operator, implementing barriers between batches, and performing the third operator;
finalizing the stored information in the staging area;
generating a materialized result file as an output file associated with the batch;
uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by the query coordinator; and
uploading a listing of the materialized result to an output shared file queue.

2. The method of claim 1 , further comprising:
generating a file registration request;
transmitting the file registration request to the query coordinator including the unique batch ID indicating that the batch has been processed; and
deleting the unique batch ID and information related to the batch stored in the staging area.

3. The method of claim 1 , wherein the first operator includes a table scan operator.

4. The method of claim 1 , wherein the third operator includes an insert operator.

5. The method of claim 1 , wherein the output file includes data corresponding only to the batch and no other batch.

6. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving an assignment of a batch an assignment of a batch from a query coordinator, the batch including a group of a set of files from a shared file queue;
storing a unique batch ID and information related to the batch in a staging area;;
performing a first operator on the batch;
based on performing the first operator, generating a checkpoint signal;
determining a state condition of a second operator as being stateless;
based on the stateless condition of the second operator, performing the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;
determining a state condition of the third operator as being non-stateless;
based on the non-stateless condition of the third operator, clearing data associated with the third operator, implementing barriers between batches, and performing the third operator;
finalizing the stored information in the staging area;
generating a materialized result file as an output file associated with the batch;
uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by the query coordinator; and
uploading a listing of the materialized result to an output shared file queue.

7. The system of claim 6 , the operations further comprising:
transmitting the output file associated with the batch;
generating a file registration request; and
deleting the unique batch ID and information related to the batch.

8. The system of claim 6 , wherein the first operator includes a table scan operator.

9. The system of claim 6 , wherein the third operator includes an insert operator.

10. The system of claim 6 , wherein the output file includes data corresponding only to the batch and no other batch.

11. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by one or more processors, an assignment of a batch from a query coordinator, the batch including a group of a set of files from a shared file queue;
storing a unique batch ID and information related to the batch in a staging area;;
performing a first operator on the batch;
based on performing the first operator, generating a checkpoint signal;
determining a state condition of a second operator as being stateless;
based on the stateless condition of the second operator, performing the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;
determining a state condition of the third operator as being non-stateless;
based on the non-stateless condition of the third operator, clearing data associated with the third operator, implementing barriers between batches, and performing the third operator;
generating a materialized result file as an output file associated with the batch;
uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by the query coordinator; and
uploading a listing of the materialized result to an output shared file queue.

12. The machine-storage medium of claim 11 , further comprising:
transmitting the output file associated with the batch;
generating a file registration request; and
deleting the unique batch ID and information related to the batch.

13. The machine-storage medium of claim 11 , wherein the first operator includes a table scan operator.

14. The machine-storage medium of claim 11 , wherein the third operator includes an insert operator.

15. The machine-storage medium of claim 11 , wherein the output file includes data corresponding only to the batch and no other batch.

Assignees

• Snowflake Inc

Inventors

• Cruanes Thierry
• Demura Igor
• Ganesh Varun
• Rajaperumal Prasanna
• Wang Libo
• Yan Jiaqi

Classifications

• G06F16/24539

  using cached or materialised query results · CPC title

• G06F16/122

  using management policies (point-in-time backing up or restoration of persistent data G06F11/1446; file migration policies for HSM systems G06F16/185) · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06F16/24542 Primary

  Plan optimisation · CPC title

• G06F9/485

  Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence) · CPC title