Optimized resource metering in a multi tenanted distributed file system

Patent metadata

Field	Value
Publication number	US-10120724-B2
Application number	US-201615237784-A
Country	US
Kind code	B2
Filing date	Aug 16, 2016
Priority date	Aug 16, 2016
Publication date	Nov 6, 2018
Grant date	Nov 6, 2018

Abstract

A method and system for automatically metering a distributed file system node is provided. The method includes receiving data associated with jobs for execution via a distributed file system. Characteristics of the jobs are uploaded and policy metrics data associated with hardware usage metering is retrieved. Resource requests associated with hardware resource usage are retrieved and attributes associated with the resource requests are uploaded. The policy metrics data is analyzed and a recommendation circuit is queried with respect to the resource requests. A set of metrics of the policy metrics data associated with the resource requests is determined and a machine learning circuit is updated. Utilized hardware resources are determined with respect to the hardware usage metering and said resource requests.

First claim

What is claimed is:

1. A distributed file system metering and hardware usage technology improvement method comprising:
receiving from a user, by a processor of a hardware device comprising specialized discrete non-generic analog, digital, and logic based plugin circuitry including a specially designed integrated circuit designed for only implementing said distributed file system metering and hardware usage technology improvement method, job data associated with jobs for execution via a distributed file system;
uploading from said job data, by said processor to a memory device of said hardware device, characteristics of said jobs;
retrieving, by said processor enabling a policy engine circuit of said hardware device, policy and cost metrics data associated with hardware usage metering, wherein said policy and cost metrics data comprises policies implemented as pluggable components defined in advance, by: uploading xml files, via descriptor files or json fides, or via a command line, for said jobs;
retrieving, by said processor enabling a hardware device cluster, resource requests describing hardware resource usage of said jobs and metered with respect to a fine grained level of the actual utilization of hardware resources;
querying, by said processor enabling a job descriptor engine of said hardware device, a recommendation circuit for locating said policy and cost metrics data of said resource requests;
determining, by said processor enabling a machine learning circuit with respect to results of said querying, a set of metrics of said policy and cost metrics data associated with said resource requests;
updating, by said processor based on said set of metrics, said machine learning circuit with characteristics of said jobs associated with network delays, sudden change of memory data volumes, and hardware device failure;
detecting, by said processor enabling voltage sensors comprised by said hardware device, voltages associated with a hardware cluster;
detecting, by said processor enabling temperature sensors comprised by said hardware device, temperature readings associated with said hardware cluster, wherein said voltages and said temperature readings are analyzed to indicate a speed and usage of CPUs and a memory space available for said hardware cluster;
determining, by said processor enabling a metrics circuit with respect to said set of metrics, said voltages, and said temperature readings, utilized hardware resources of said hardware cluster with respect to said hardware usage metering and said resource requests; and
allocating, by said processor based on results of said characteristics of said jobs and results of said determining said utilized hardware resources, specified queue memory and associated processor cores to said hardware cluster thereby enabling control functionality for overall resource utilization of said hardware cluster and improving execution of said jobs being executed on said hardware cluster;
generating, by said processor executing a resource descriptor circuit, resource infrastructure associated with hardware resources of said resource requests, wherein said generating is executed based on said policy engine circuit providing various policies and cost metrics for calculating said hardware usage metering, and wherein said policy engine circuit comprises the pluggable components;
monitoring, by said processor executing said job descriptor engine, a plurality of nodes, hardware resources, processors, memory devices, and distributed system networks; and
determining, by said processor, a configuration of said hardware device cluster based on an analysis of types of workloads and job characterizations;
determining, by said processor, workloads associated with said plurality of nodes, said hardware resources, said processors, said memory devices, and said distributed system networks;
determining, by said processor, a recommendation for improving said configuration of said hardware device cluster; and
calculating, by said computer processor, a cost for improving said configuration of said hardware device cluster, wherein said

Cost
=

(

Confidence
⁢

⁢
Factor

)

*

∑
0
n

⁢

(

(

a
⁢

∑

0
⁢
%

100
⁢
%

⁢
Memory

)

+

(

b
⁢

∑

0
⁢
%

100
⁢
%

⁢

CPU
⁢

⁢
Utilization

)

+

(

c
⁢

∑
1
∞

⁢

Network
⁢

⁢
Data
⁢

⁢
Transfer

)

+

Assignees

• IBM

Inventors

• Badjatia Sudeep
• Bhose Rajarshi
• Deb Bijoy
• Nandi Ripon

Classifications

• H04M15/31

  Distributed metering or calculation of charges · CPC title

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• G06F11/3442

  for planning or managing the needed capacity · CPC title

• G06F11/3447

  Performance evaluation by modeling · CPC title

• H04M15/66

  Policy and charging system · CPC title