Resolving container preemption

Patent metadata

Field	Value
Publication number	US-11150956-B2
Application number	US-201916417678-A
Country	US
Kind code	B2
Filing date	May 21, 2019
Priority date	May 21, 2019
Publication date	Oct 19, 2021
Grant date	Oct 19, 2021

Abstract

A set of resources required to process a data integration job is determined. In response to determining that the set of resources is not available, queue occupation, for each queue in the computing environment, is predicted. Queue occupation is a workload of queue resources for a future time based on a previous workload. A best queue is selected based on the predicted queue occupation. The best queue is the queue or queues in the computing environment available to be assigned to process the data integration job without preemption. The data integration job is processed using the best queue. It is determined whether a preemption event occurred causing the removal of resources from the best queue. A checkpoint is created in response to determining that a preemption event occurred. The checkpoint indicates the last successful operation completed and provides a point where processing can resume when resources become available.

First claim

What is claimed is:

1. A method, the method comprising:
determining, by one or more computer processors, a set of resources within a computing environment required to process a data integration job to completion;
determining, by one or more computer processors, whether the set of resources are available to process the data integration job to completion;
responsive to determining that the set of resources are not available, predicting, by one or more computer processors, a queue occupation for each queue of one or more queues in the computing environment, wherein;
the queue occupation is a workload of resources within each queue of the one or more queues for a future time period based on a previous workload history of said resources, and
each queue of the one or more queues in the computing environment include a portion of total resources within the computing environment;

selecting, by one or more computer processors, a best queue of the one or more queues to process the data integration job based on the predicted queue occupation, wherein the best queue is the one or more queues within the computing environment available, based on the predicted queue occupation, to be assigned to one or more containers that include at least the determined set of resources required to process the data integration job without a preemption event occurring;
processing, by one or more computer processors, the data integration job using the selected best queue;
determining, by one or more computer processors, whether the preemption event causing removal of resources from the best queue processing the data integration job has occurred, wherein removal of the resources preempts processing of the data integration job; and
responsive to determining that the preemption event did occur, creating, by one or more computer processors, a checkpoint, wherein;
the checkpoint indicates a last successful operation completed while processing the data integration job, and
the checkpoint provides a point where processing of the data integration job can resume when enough resources become available.

2. The method of claim 1 , further comprising:
responsive to creating the checkpoint, exiting, by one or more computer processors, the processing of the data integration job;
determining, by one or more computer processors, for a first time whether resources are available to continue processing the data integration job, wherein the determination is made after a first period of time; and
responsive to determining for the first time that resources are available to continue processing the data integration job, processing, by one or more computer processors, the data integration job starting at the created checkpoint.

3. The method of claim 2 , further comprising:
responsive to determining for the first time that resources are not available to continue processing the data integration job, determining, by one or more computer processors, for a second time whether resources are available to continue processing the data integration job, wherein the determination is made after a second period of time.

4. The method of claim 1 , further comprising:
responsive to determining that the preemption event has not occurred, completing, by one or more computer processors, the processing of the data integration job, and
reporting, by one or more computer processors, a result of the processing of the data integration job.

5. The method of claim 1 , wherein the computing environment includes a plurality of containers assigned to the one or more queues in the computing environment and each container in the plurality of containers is assigned and utilizes a portion of an overall processing capability of the computing environment and a portion of an overall memory capability of the computing environment.

6. The method of claim 1 , wherein the preemption event occurs when computing resources assigned to a first queue, in use by a second queue, are required by the first queue for other processing and are subsequently removed from the second queue preempting processing in the second queue.

7. The method of claim 1 , wherein the set of resources includes a time duration required to process the data integration job to completion.

8. A computer program product, the computer program product comprising:
one or more computer readable storage media, and
program instructions stored on the one or more computer readable storage media, the program instructions comprising:
program instructions to determine a set of resources within a computing environment required to process a data integration job to completion;
program instructions to determine whether the set of resources are available to process the data integration job to completion;
responsive to determining that the set of resources are not available, program instructions to predict a queue occupation for each queue of one or more queues in the computing environment, wherein;
the queue occupation is a workload of resources within each queue of the one or more queues for a future time period based on a previous workload history of said resources, and
each queue of the one or more queues in the computing environment include a portion of total resources within the computing environment;

program instructions to select a best queue of the one or more queues to process the data integration job based on the predicted queue occupation, wherein the best queue is the one or more queues within the computing environment available, based on the predicted queue occupation, to be assigned to one or more containers that include at least the determined set of resources required to process the data integration job without a preemption event occurring;
program instructions to process the data integration job using the selected best queue;
program instructions to determine whether the preemption event causing removal of resources from the best queue processing the data integration job has occurred, wherein removal of the resources preempts processing of the data integration job; and
responsive to determining that the preemption event did occur, program instructions to create a checkpoint, wherein;
the checkpoint indicates a last successful operation completed while processing the data integration job, and
the checkpoint provides a point where processing of the data integration job can resume when enough resources become available.

9. The computer program product of claim 8 , further comprising program instructions stored on the one or more computer readable storage media, to:
responsive to creating the checkpoint, exit the processing of the data integration job;
determine for a first time whether resources are available to continue processing the data integration job, wherein the determination is made after a first period of time; and
responsive to determining for the first time that resources are available to continue processing the data integration job, process the data integration job starting at the created checkpoint.

10. The computer program product of claim 9 , further comprising program instructions stored on the one or more computer readable storage media, to:
responsive to determining for the first time that resources are not available to continue processing the data integration job, determine for a second time whether resources are available to continue processing the data integration job, wherein the determination is made after a second period of time.

11. The computer program product of claim 8 , further comprising program instructions stored on the one or more computer readable storage media, to:
responsive to determining that the preemption event has not occurred, complete the proc

Assignees

• IBM

Inventors

• Bonagiri Krishna Kishore
• Jacobson Eric A
• Gupta Ritesh Kumar
• Brokaw Scott Louis

Classifications

• G06F9/5027 Primary

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06F9/461

  Saving or restoring of program or task context · CPC title

• G06F9/505 Primary

  considering the load · CPC title

• G06F9/4887

  involving deadlines, e.g. rate based, periodic · CPC title

• G06F2209/5019

  Workload prediction · CPC title