System and method of providing system jobs within a compute environment

Patent metadata

Field	Value
Publication number	US-11494235-B2
Application number	US-202217700808-A
Country	US
Kind code	B2
Filing date	Mar 22, 2022
Priority date	Nov 8, 2004
Publication date	Nov 8, 2022
Grant date	Nov 8, 2022

Abstract

The disclosure relates to systems, methods and computer-readable media for using system jobs for performing actions outside the constraints of batch compute jobs submitted to a compute environment such as a cluster or a grid. The method for modifying a compute environment from a system job disclosure associating a system job to a queuable object, triggering the system job based on an event and performing arbitrary actions on resources outside of compute nodes in the compute environment. The queuable objects include objects such as batch compute jobs or job reservations. The events that trigger the system job may be time driven, such as ten minutes prior to completion of the batch compute job, or dependent on other actions associated with other system jobs. The system jobs may be utilized also to perform rolling maintenance on a node by node basis.

First claim

What is claimed is:

1. A method of configuring at least a portion of a multi-node compute environment, the method comprising:
generating, by a computer processor, a system job in the multi-node compute environment, the system job which configures at least one aspect of the compute environment for subsequent utilization by one or more compute jobs without the system job executing any workload associated with the one or more compute jobs; and
performing, by the computer processor, the configuration of the at least one aspect of the compute environment by processing the system job,

wherein:
the system job further configures the at least one aspect of the compute environment for subsequent utilization by the one or more compute jobs without the system job making any kind of reservation or allocation of resources for the one or more compute jobs.

2. The method of claim 1 , wherein:
the subsequent utilization by the one or more compute jobs comprises consumption of compute resources within the multi-node compute environment.

3. The method of claim 1 , further comprising associating the generated system job with one or more queued or queueable workload items of the one or more compute jobs, the associating comprising enabling one or more monitored conditions relating to the one or more queued or queueable workload items to trigger one or more actions of the generated system job.

4. The method of claim 3 , wherein:
the system job further configures the at least one aspect of the compute environment for subsequent utilization by the one or more compute jobs without the system job making any allocation of the one or more compute jobs or portions thereof, to one or more nodes of the multi-node compute environment for execution; and
the subsequent utilization by the one or more compute jobs comprises consumption of compute resources within the multi-node compute environment.

5. The method of claim 1 , wherein:
configuration by the system job is triggered by an event associated with a second system job; and
the configuration of the at least one aspect of the compute environment comprises performance of a rolling maintenance task.

6. The method of claim 5 , wherein the event associated with a second system job comprises the start of an application within the multi-node compute environment.

7. The method of claim 1 , wherein the configuration of the at least one aspect of the compute environment comprises configuration of a storage resource outside of compute nodes of the multi-node compute environment.

8. The method of claim 1 , wherein the configuration of the at least one aspect of the compute environment comprises performance of a rolling maintenance task, and the performing the configuration of the at least one aspect of the compute environment by processing the system job comprises iteratively processing the system job on a node-by-node basis in order to effect the rolling maintenance task on a plurality of the nodes of the multi-node compute environment.

9. The method of claim 8 , wherein the iteratively processing the system job on a node-by-node basis in order to affect the rolling maintenance task on a plurality of the nodes of the multi-node compute comprises processing the system job on a node-by-node-bases to effect at least one of a software update or operating system configuration process.

10. The method of claim 1 , wherein the method further comprises:
determining that one or more resources that are required for processing the system job have become available; and
causing the processing the job based on the determining.

11. The method of claim 1 , wherein the configuration of the at least one aspect of the compute environment for subsequent utilization comprises creating at least one virtual cluster within the multi-node compute environment, the virtual cluster dedicated to at least one of (i) one or more particular workload tasks or groups of tasks, or (ii) one or more particular users or groups of users.

12. The method of claim 11 , wherein the configuration of the at least one aspect of the compute environment for subsequent utilization further comprises automatically de-establishing or terminating the at least one virtual cluster within the multi-node compute environment after one or more prescribed conditions are met.

13. The method of claim 1 , wherein the system job comprises a queueable object which can be submitted to a scheduling environment of a scheduler process associated with the multi-node compute environment, but which can affect one or more configurations of the multi-node compute environment which are outside of the scheduling environment.

14. The method of claim 1 , wherein the system job has one or more dependencies, the one or more dependencies based on a type of resources of the multi-node compute environment.

15. The method of claim 1 , wherein the system job has one or more content-sensitive triggers, the one or more content-sensitive triggers enabling the system job to customize at least a portion of the multi-node compute environment.

16. The method of claim 1 , wherein the system job has one or more content-sensitive triggers, the one or more content-sensitive triggers enabling the system job to customize at least a scheduling environment of a scheduler process associated with the multi-node compute environment.

17. The method of claim 1 , wherein the system job comprises a plurality of steps, at least one of the plurality of steps configured to dynamically allocate one or more resources internal to the multi-node compute environment based at least on a result of a previous one of the plurality of steps.

18. The method of claim 1 , wherein the system job comprises a plurality of steps, at least one of the plurality of steps configured to dynamically allocate one or more resources external to the multi-node compute environment, the allocated one or more resources comprising resources not considered by a computerized scheduler process which manages resource scheduling for resources within the multi-node compute environment.

19. The method of claim 1 , wherein the system job comprises a plurality of steps, at least one of the plurality of steps configured to cause enveloping or inclusion of, a plurality of compute tasks associated within a first layer of abstraction, within a logical process disposed at a layer of abstraction higher than the first layer.

20. The method of claim 1 , wherein the system job comprises a plurality of steps, at least one of the plurality of steps configured to cause synchronization of at least two tasks, the at least two tasks each managing one or more resources or resource types that are unrelated to one or more others of the at least two tasks.

21. The method of claim 1 , wherein the generating the system job comprises generating the system job responsive to submission of compute workload to the multi-node compute environment, the submission of the compute workload specifying at least one quality of service (QoS) requiring enablement of at least one special service of the multi-node compute environment.

22. The method of claim 21 , wherein the at least one special service of the multi-node compute environment comprises a service which automatically provisions one or more resources of the multi-node compute environment, the provisioned one or more resources necessary to support the at least one QoS.

23. The method of claim 21 , wherein the at least one special service of the multi-node compute

Assignees

• Iii Holdings 12 Llc

Inventors

• Jackson David B

Classifications

• G06F8/61

  Installation · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/5011 Primary

  the resources being hardware resources other than CPUs, Servers and Terminals · CPC title

• G06F9/542

  Event management; Broadcasting; Multicasting; Notifications · CPC title