Enhanced configuration management of data processing clusters

Patent metadata

Field	Value
Publication number	US-12306735-B2
Application number	US-202318341325-A
Country	US
Kind code	B2
Filing date	Jun 26, 2023
Priority date	Apr 5, 2019
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

Described herein are systems, methods, and software to enhance the management and deployment of data processing clusters in a computing environment. In one example, a management system may monitor data processing efficiency information for a cluster and determine when the efficiency meets efficiency criteria. When the efficiency criteria are met, the management system may identify a new configuration for the cluster and initiate an operation to implement the new configuration for the cluster.

First claim

What is claimed is:

1. A method comprising:
monitoring, by a computing system, data processing efficiency information for a cluster deployed with a first configuration in a computing environment;
determining, by the computing system, when the data processing efficiency information meets at least one criterion to transition the cluster from the first configuration to a second configuration by determining that a first data processing efficiency score associated with the first configuration fails to satisfy a threshold efficiency value;
when the data processing efficiency information meets the at least one criterion, identifying, by the computing system, one or more suggested configuration modifications for the cluster;
generating, for display, a notification to indicate the one or more suggested configuration modifications;
identifying a selection of at least one modification for the cluster from the one or more suggested configuration modifications; and
initiating, by the computing system, implementation of the at least one modification to transition the cluster from the first configuration to the second configuration.

2. The method of claim 1 , wherein the data processing efficiency information comprises a data processing rate from a storage repository.

3. The method of claim 2 , wherein the storage repository comprises a distributed file system or object storage system.

4. The method of claim 2 , wherein the data processing efficiency information comprises physical resource allocation information for the cluster.

5. The method of claim 1 , wherein the data processing efficiency information comprises a score indicative of a relationship between a data processing rate of the cluster to a physical resource allocation to the cluster.

6. The method of claim 1 , wherein the second configuration comprises one or more different configuration attributes than the first configuration, wherein the one or more different configuration attributes comprise processing resources for virtual or physical nodes of the cluster, memory resources for virtual nodes of the cluster, a data processing service version for the cluster, or a quantity of virtual nodes in the cluster.

7. The method of claim 1 , wherein identifying the one or more suggested configuration modifications for the cluster comprises:
identifying a data processing service associated with the cluster;
identifying one or more other clusters executing the data processing service;
determining data processing efficiency information associated with the one or more other clusters; and
identifying the one or more suggested configuration modifications for the cluster based on the data processing efficiency information associated with the one or more other clusters and configuration differences between the cluster and the one or more other clusters.

8. The method of claim 1 , wherein the cluster comprises a plurality of containers.

9. The method of claim 1 , wherein identifying the one or more suggested configuration modifications for the cluster comprises:
identifying data processing efficiency information associated with one or more previous configurations of the cluster; and
identifying the one or more suggested configuration modifications for the cluster based on the data processing efficiency information associated with the one or more previous configurations of the cluster and configuration differences between the first configuration of the cluster and the one or more previous configurations of the cluster.

10. An apparatus comprising:
one or more non-transitory computer readable storage media;
a processing system operatively coupled to the one or more non-transitory computer readable storage media; and
program instructions stored on the one or more non-transitory computer readable storage media that, when executed by the processing system, direct the processing system to:
monitor data processing efficiency information for a cluster deployed with a first configuration in a computing environment, wherein the cluster comprises a plurality of containers;
determine when the data processing efficiency information meets at least one criterion to transition the cluster from the first configuration to a second configuration by determining that a first data processing efficiency score associated with the first configuration fails to satisfy a threshold efficiency value;
when the data processing efficiency information meets the at least one criterion, identify one or more suggested configuration modifications for the cluster;
generate, for display, a notification to indicate the one or more suggested configuration modifications;
identify a selection of at least one modification for the cluster from the one or more suggested configuration modifications; and
initiate implementation of the at least one modification to transition the cluster from the first configuration to the second configuration.

11. The apparatus of claim 10 , wherein the data processing efficiency information comprises a data processing rate from a storage repository.

12. The apparatus of claim 11 , wherein the data processing efficiency information comprises physical resource allocation information for the cluster.

13. The apparatus of claim 10 , wherein the data processing efficiency information comprises a score indicative of a relationship between a data processing rate of the cluster to a physical resource allocation to the cluster.

14. The apparatus of claim 10 , wherein the program instructions that direct the processing system to identify the one or more suggested configuration modifications for the cluster include instructions that direct the processing system to:
identify a data processing service associated with the cluster;
identify one or more other clusters executing the data processing service;
determine data processing efficiency information associated with the one or more other clusters; and
identify the one or more suggested configuration modifications for the cluster based on the data processing efficiency information associated with the one or more other clusters and configuration differences between the cluster and the one or more other clusters.

15. The apparatus of claim 10 , wherein the program instructions that direct the processing system to identify the one or more suggested configuration modifications for the cluster include instructions that direct the processing system to:
identify data processing efficiency information associated with one or more previous configurations of the cluster; and
identify the one or more suggested configuration modifications based on the data processing efficiency information associated with the one or more previous configurations of the cluster and configuration differences between the first configuration of the cluster and the one or more previous configurations of the cluster.

16. The apparatus of claim 15 , wherein the plurality of containers executes a distributed data processing service to process data from a storage repository comprising a distributed file system.

17. A method comprising:
monitoring data processing efficiency information for a cluster deployed with a first configuration in a computing environment;
determining when the data processing efficiency information meets at least one criterion to transition the cluster from the first configuration to a second configuration by determining that a first data processing efficiency score associated with the first configuration fails to satisfy a threshold efficiency value;
when the data processing efficiency information meets th

Assignees

• Hewlett Packard Entpr Dev Lp

Inventors

• Baxter Joel

Classifications

• G06F11/3428

  Benchmarking · CPC title

• G06F11/3433

  for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029) · CPC title

• G06F11/3404

  for parallel or distributed programming · CPC title

• G06F11/3048

  where the topology of the computing system or computing system component explicitly influences the monitoring activity, e.g. serial, hierarchical systems · CPC title

• G06F11/3409

  for performance assessment · CPC title