Processing of long running processes

Patent metadata

Field	Value
Publication number	US-9703594-B1
Application number	US-201514635254-A
Country	US
Kind code	B1
Filing date	Mar 2, 2015
Priority date	Mar 2, 2015
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

A system adapted to process long-running processes is disclosed. A request to upload data is received at a server. The server divides the data into multiple parts and launches a separate process to upload each of the divided parts. The server records for each process the processing time or duration that the particular process used to upload its corresponding data item. The server maintains an average processing duration that is calculated from the processing durations of the completed processes. The server identifies that one process is continuing to run and compares a processing duration for the particular process to a threshold derived from the average processing duration. If the processing duration for the particular process exceeds the threshold, the server initiates a new process to upload the same data item. When one of either the new process or the still running process has completed processing, the server terminates the other process.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a request to upload a data item;
dividing the data item into a plurality of data parts, each of the data parts representing part of the data item;
initiating a plurality of processes, each of the plurality of processes performing processing to upload a respective one of the plurality of data parts;
recording a processing duration for each process of the plurality of processes that completes processing to upload its respective data part representing part of the data item;
determining at least one of the plurality of processes continues processing to upload its respective data part representing part of the data item;
for each of the at least one of the plurality of processes continuing processing to upload its respective data part representing part of the data item, comparing a processing duration to a threshold derived from an average processing duration for processes that have completed processing to upload their respective data parts representing parts of the data item; and
for each of the at least one of the plurality of processes continuing processing to upload its respective data part representing part of the data item, initiating a new process to upload the respective data part upon determining the processing duration exceeds the threshold.

2. The method of claim 1 , further comprising:
for each new process initiated to upload the data part upon determining the processing duration exceeds the threshold:
determining the new process or the one of the plurality of processes continuing processing to upload the respective data part has completed processing to upload the respective data part; and
terminating processing of the new process or the one of the plurality of processes continuing processing to upload the respective data part.

3. The method of claim 1 , wherein dividing the data item into a plurality of data parts comprises dividing the data item into equal data parts.

4. The method of claim 1 , wherein recording a processing duration for each process of the plurality of processes that completes processing to upload a data part representing part of the data item, comprises:
recording, for each process, a start time identifying when processing started;
recording, for each process, a completion time identifying when processing completed; and
calculating, for each process, a difference between the completion time and the start time.

5. A computing system for uploading data, comprising:
computing memory having stored therein computer instructions that, upon execution by one or more processors, at least cause the system to perform operations comprising:
initiating a plurality of processes, each of the plurality of processes performing processing to upload a respective data item, each respective data item representing part of a whole data item;
recording a processing duration for each process of the plurality of processes that finishes uploading its respective data item representing part of the whole data item;
determining a process of the plurality of processes is continuing to upload its respective data item representing part of the whole data item;
comparing a processing duration of the process continuing to upload its respective data item representing part of the whole data item to a duration threshold derived from processing durations recorded for processes that have finished uploading their respective data item representing part of the whole data item; and
based upon determining the process duration of the process continuing to upload its respective data item representing part of the whole data item exceeds the duration threshold derived from the process durations recorded for processes that have finished uploading their respective data item representing part of the whole data item, initiating a new process to upload the respective data item of the process continuing to upload.

6. The system of claim 5 , the operations further comprising:
receiving a request to upload the whole data item; and
dividing the whole data item into the data items.

7. The system of claim 6 , wherein the data items are of equal size.

8. The system of claim 6 , wherein recording an uploading duration for each process that finishes uploading its respective data item representing part of the whole data item, comprises:
recording, for each process, a start time identifying when uploading started;
recording, for each process, a completion time identifying when uploading finished; and
calculating, for each process, a difference between the completion time and the start time.

9. The system of claim 6 , wherein receiving a request to upload the whole data item comprises receiving a request to upload the whole data item via a web service.

10. The system of claim 5 , the operations further comprising:
calculating an average computing duration reflecting computing durations for each process that finishes uploading its respective data item, and wherein the duration threshold is derived from the average computing duration.

11. The system of claim 10 , wherein the duration threshold is the average computing duration, 1.1 times the average computing duration, 1.2 times the average computing duration, 1.3 times the average computing duration, 1.4 times the average computing duration, or 1.5 times the average computing duration.

12. The system of claim 5 , wherein determining the process of the plurality of processes continuing to upload its respective data item comprises determining that a number of the plurality of processes that have finished uploading exceeds a preliminary threshold.

13. The system of claim 5 , the operations further comprising:
determining that one of the new process and the process continuing to upload the respective data item has finished uploading the respective data item; and
terminating uploading of the other one of the new process and the process continuing to upload the respective data item.

14. The system of claim 5 ,
wherein the duration threshold is derived by multiplying an average computing duration by a multiple that varies depending upon the number of processes that have finished uploading their respective data items representing part of the whole data item.

15. A non-transitory computer-readable storage medium having stored thereon computer instructions that, upon execution by one or more processors, at least cause a computing system to perform operations comprising:
dividing a data item into at least a first data part corresponding to part of the data item, a second data part corresponding to part of the data item, and a third data part corresponding to part of the data item;
initiating a first process to upload the first data part, a second process to upload the second data part, and a third process to upload the third data part;
upon the first process completing uploading the first data part, recording a first duration associated with uploading the first data part;
upon the second process completing uploading the second data part, recording a second duration associated with uploading the second data part;
determining a processing duration of the third process exceeds a duration threshold formed using at least the first duration and the second duration; and
based upon determining the process duration of the third process exceeds the duration threshold formed using at least the first duration and the second duration, initiating a new process to upload the third data part.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the duration threshold is

Assignees

• Amazon Tech Inc

Inventors

• Kamboj Ankit
• Wu Xing
• McPherson George Steven
• Fang Jian
• Stockstad Dag
• Sinha Abhishek Rajnikant

Classifications

• G06F9/4806 Primary

  Task transfer initiation or dispatching · CPC title

• G06F9/4887

  involving deadlines, e.g. rate based, periodic · CPC title

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F9/485 Primary

  Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence) · CPC title

• G06F11/3409

  for performance assessment · CPC title