File processing workflow management

Patent metadata

Field	Value
Publication number	US-2016371122-A1
Application number	US-201615182332-A
Country	US
Kind code	A1
Filing date	Jun 14, 2016
Priority date	Jun 19, 2015
Publication date	Dec 22, 2016
Grant date	—

Abstract

The present subject matter relates to management of a file processing workflow over processing instances. In an example implementation, a notification message is received from a producer-processing instance of a cluster of processing instances. The notification message comprises a cluster identifier of the cluster associated with the producer-processing instance, and a data resource identifier associated with output data produced by the producer-processing instance in a file processing workflow. Further, the notification message received from the producer-processing instance is forwarded based on the cluster identifier to a message queue for the cluster maintained in the computer. The notification message is transmitted from the message queue for the cluster to a consumer-processing instance of the cluster for processing the output data based on the data resource identifier.

First claim

We claim:

1 . A method for managing a file processing workflow over multiple processing instances, the method comprising:
maintaining, by a computing system, a message queue for a cluster of processing instances, the cluster of processing instances comprising at least one producer-processing instance and at least one consumer-processing instance; receiving, by the computing system from a producer-processing instance of the cluster, a notification message comprising a cluster identifier of the cluster associated with the producer-processing instance, and a data resource identifier associated with output data produced by the producer-processing instance in the file processing workflow; forwarding, by the computing system, the notification message received from the producer-processing instance to the message queue for the cluster based on the cluster identifier; and transmitting, by the computing system, the notification message from the message queue for the cluster to a consumer-processing instance of the cluster for processing the output data based on the data resource identifier.

2 . The method as claimed in claim 1 , wherein the cluster identifier is a cluster name, and wherein the data resource identifier is a uniform resource locator (URL) for the output data.

3 . The method as claimed in claim 1 , wherein the cluster is one of a plurality of clusters of processing instances in a multi-tier processing instance environment, and wherein the at least one producer-processing instance and the at least one consumer-processing instance of the cluster are at different tiers of the multi-tier processing instance environment.

4 . The method as claimed in claim 1 , wherein the transmitting the notification message to the consumer-processing instance of the cluster is based on receiving a job request message from the consumer-processing instance of the cluster.

5 . The method as claimed in claim 1 , comprising:
determining, by the computing system, a number of pending jobs for the cluster based on a number of notification messages pending in the message queue for the cluster; and registering, by the computing system, in real-time, at least one new consumer-processing instance for the cluster when the number of pending jobs exceeds a pending jobs threshold value.

6 . The method as claimed in claim 1 , comprising:
determining, by the computing system, a job production throughput for the cluster based on a rate at which notification messages are forwarded in the message queue for the cluster; determining, by the computing system, a job consumption throughput for the cluster based on a rate at which notification messages are consumed from the message queue for the cluster; registering, by the computing system, in real-time, at least one new consumer-processing instance for the cluster when the job production throughput exceeds the job consumption throughput by a first threshold rate value; and deregistering, by the computing system, in real-time, the at least one consumer-processing instance for the cluster when the job consumption throughput exceeds the job production throughput by a second threshold rate value.

7 . The method as claimed in claim 1 , comprising:
receiving, by the computing system, meta-data associated with the output data from the at least one producer-processing instance of the cluster; and providing, by the computing system, the meta-data to an audit manager for auditing the file processing workflow.

8 . A system for managing a file processing workflow over multiple processing instances, the system comprising:
a processor; a message broker coupled to the processor to:
register at least one producer-processing instance and at least one consumer-processing instance in a cluster of processing instances;
maintain a message queue for the cluster of processing instances;
receive from a producer-processing instance of the cluster a notification message comprising a cluster identifier of the cluster associated with the producer-processing instance, and a data resource identifier associated with output data produced by the producer-processing instance in the file processing workflow; and
forward the notification message received from the producer-processing instance to the message queue for the cluster based on the cluster identifier; and
a job dispatcher coupled to the processor to transmit the notification message from the message queue for the cluster to a consumer-processing instance of the cluster for processing the output data based on the data resource identifier.

9 . The system as claimed in claim 8 , wherein the cluster identifier is a cluster name, and wherein the data resource identifier is a uniform resource locator (URL) for the output data.

10 . The system as claimed in claim 8 , comprising a job monitor coupled to the processor to determine a number of pending jobs for the cluster based on a number of notification messages pending in the message queue for the cluster, wherein the message broker is to register, in real-time, at least one new consumer-processing instance for the cluster when the number of pending jobs exceeds a pending jobs threshold value.

11 . The system as claimed in claim 8 , comprising a job monitor coupled to the processor to:
determine a job production throughput for the cluster based on a rate at which notification messages are forwarded in the message queue for the cluster; determine a job consumption throughput for the cluster based on a rate at which notification messages are consumed from the message queue for the cluster; and wherein the message broker is to register, in real-time, at least one new consumer-processing instance for the cluster when the job production throughput exceeds the job consumption throughput by a first threshold rate value.

12 . The system as claimed in claim 8 , comprising a job monitor coupled to the processor to:
determine a job production throughput for the cluster based on a rate at which notification messages are forwarded in the message queue for the cluster; determine a job consumption throughput for the cluster based on a rate at which notification messages are consumed from the message queue for the cluster; and wherein the message broker is to deregister, in real-time, the at least one consumer-processing instance for the cluster when the job consumption throughput exceeds the job production throughput by a second threshold rate value.

13 . A non-transitory computer-readable medium comprising computer-readable instructions, which, when executed by a computer, cause the computer to:
receive from a producer-processing instance of a cluster of processing instances a notification message comprising a cluster identifier of the cluster associated with the producer-processing instance, and a data resource identifier associated with output data produced by the producer-processing instance in a file processing workflow; forward, based on the cluster identifier, the notification message received from the producer-processing instance to a message queue for the cluster maintained in the computer; and transmit the notification message from the message queue for the cluster to a consumer-processing instance of the cluster for processing the output data based on the data resource identifier.

14 . The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions which, when executed by the computer, cause the computer to:
determine a number of pending jobs for the cluster based on a number of notification messages pending in the message queue for the cluster; and register, with the compu

Assignees

• Hewlett Packard Entpr Dev Lp

Inventors

• Nair Rajesh P

Classifications

• G06F9/46

  Multiprogramming arrangements · CPC title

• G06F9/546 Primary

  Message passing systems or structures, e.g. queues · CPC title

• G06F9/505 Primary

  considering the load · CPC title

• G06F9/542

  Event management; Broadcasting; Multicasting; Notifications · CPC title

• G06F9/5005

  to service a request · CPC title