File operation task optimization

Patent metadata

Field	Value
Publication number	US-11036547-B2
Application number	US-201916451829-A
Country	US
Kind code	B2
Filing date	Jun 25, 2019
Priority date	Nov 25, 2013
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

A method includes receiving, by a data processing apparatus, a plurality of file operation requests, each file operation request including a priority, a deadline, and an operation type and representing a request to perform an operation on at least one file maintained in a distributed file system; identifying, by the data processing apparatus, a group of file operation requests to be executed together from the plurality of file operation requests, the identification based at least in part on at least one of: the file operations in the group of file operations being directed to a same storage system, or file operations in the group of file operations sharing a common operation type; and sending a request to execute the group of file operation requests to a system configured to perform the group of file operation requests.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a data processing apparatus, a plurality of file operation requests, each file operation request representing a request to perform an operation on at least one file maintained in a distributed file system and corresponding to a priority and to an operation type;
selecting, by the data processing apparatus, a particular file operation request of the plurality of file operation requests based on the priority of the particular file operation request relative to the priorities of the plurality of file operation requests;
in response to selecting the particular file operation request of the plurality of file operation requests based on the priority of the particular file operation request relative to the priorities of the plurality of file operation requests, selecting, by the data processing apparatus, a group of file operation requests to be performed together with the particular file operation request, the group of file operation requests having a common operation type with the operation type of the selected particular file operation request; and
sending, by the data processing apparatus, a single request to perform both the selected group of file operation requests and the selected particular file operation request.

2. The method of claim 1 , further comprising, prior to sending the single request:
determining that the group of file operation requests having the common operation type with the operation type of the particular file operation request is less than a predetermined number of file operation requests; and
in response to determining that the group of file operation requests having the common operation type with the operation type of the particular file operation request is less than the predetermined number of file operation requests, waiting to receive additional file operation requests having the common operation type with the operation type of the particular file operation request.

3. The method of claim 1 , wherein at least one file operation request of the plurality of file operation requests comprises a hard deadline.

4. The method of claim 3 , further comprising aborting the at least one file operations request of the plurality of file operation requests when the at least one file operation request is not completed by the hard deadline.

5. The method of claim 3 , wherein selecting the particular file operation request of the plurality of file operation requests based on the priority of the particular file operation request relative to the priorities of the plurality of file operation requests comprises:
ordering the plurality of file operation requests in ascending order according to a timestamp for each file operation request in the plurality of file operation requests;
promoting the at least one file operation request of the plurality of file operation requests including the hard deadline to the beginning of the order of the plurality of file operation requests; and
selecting the highest priority request of the plurality of file operation requests as the particular file operation request.

6. The method of claim 3 , further comprising:
identifying a file operation request that will not be completed by a soft deadline associated with the identified file operation request;
notifying a requestor associated with the identified file operation request that the identified file operation request will not be completed before the soft deadline; and
continuing the at least one file operation request of the plurality of file operation requests with the hard deadline when the operation associated with the identified file operation request is not completed by the soft deadline associated with the identified file operation request.

7. The method of claim 1 , further comprising:
determining that a particular process associated with a file operation request of the plurality of file operation requests has terminated; and
sending a request to execute the file operation request associated with the particular process.

8. The method of claim 1 , wherein sending the single request including the group of file operation requests and the particular file operation request further comprises sending a remote procedure call.

9. The method of claim 1 , wherein the operation type comprises one or more of a file copy, a file deletion, or a file merge.

10. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving a plurality of file operation requests, each file operation request representing a request to perform an operation on at least one file maintained in a distributed file system and corresponding to a priority and to an operation type;
selecting a particular file operation request of the plurality of file operation requests based on the priority of the particular file operation request relative to the priorities of the plurality of file operation requests;
in response to selecting the particular file operation request of the plurality of file operation requests based on the priority of the particular file operation request relative to the priorities of the plurality of file operation requests, selecting a group of file operation requests to be performed together with the particular file operation request, the group of file operation requests having a common operation type with the operation type of the selected particular file operation request; and
sending a single request to perform both the selected group of file operation requests and the selected particular file operation request.

11. The system of claim 10 , wherein the operations further comprise, prior to sending the single request:
determining that the group of file operation requests having the common operation type with the operation type of the particular file operation request is less than a predetermined number of file operation requests; and
in response to determining that the group of file operation requests having the common operation type with the operation type of the particular file operation request is less than the predetermined number of file operation requests, waiting to receive additional file operation requests having the common operation type with the operation type of the particular file operation request.

12. The system of claim 10 , wherein at least one file operation request of the plurality of file operation requests comprises a hard deadline.

13. The system of claim 12 , wherein the operations further comprise
aborting the at least one file operations request of the plurality of file operation requests when the at least one file operation request is not completed by the hard deadline.

14. The system of claim 12 , wherein selecting the particular file operation request of the plurality of file operation requests based on the priority of the particular file operation request relative to the priorities of the plurality of file operation requests comprises:
ordering the plurality of file operation requests in ascending order according to a timestamp for each file operation request in the plurality of file operation requests;
promoting the at least one file operation request of the plurality of file operation requests including the hard deadline to the beginning of the order of the plurality of file operation requests; and
selecting the highest priority request of the plurality of file operation requests as the particular file operation request.

15. The system of clai

Assignees

• Google Llc

Inventors

• Ma Chi
• Goldman Kenneth J
• Zhao Yonggang
• Gildea Stephen P G

Classifications

• G06F16/182 Primary

  Distributed file systems · CPC title

• G06F9/4887

  involving deadlines, e.g. rate based, periodic · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F16/16

  File or folder operations, e.g. details of user interfaces specifically adapted to file systems · CPC title