Processes and systems for decentralization of data produced at the edge of a distributed computing system

Patent metadata

Field	Value
Publication number	US-12174875-B2
Application number	US-202217834152-A
Country	US
Kind code	B2
Filing date	Jun 7, 2022
Priority date	Jun 7, 2022
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

Computer-implemented processes and systems described herein are directed to reducing volumes of data sent from edge devices to a data center. Each edge device runs an agent that collects event information generated by event sources of the edge device in a runtime interval. Each agent reduces the event information to relevant event information at the edge device in accordance with instructions received from a controller server of the data center. The relevant event information contains less information than the event information. Each agent forwards the relevant event information over the internet to external services executed at the data center, where the relevant event information is stored in a data storage device of the data center.

First claim

The invention claimed is:

1. A computer implemented agent executed in an edge device of a distributed computing system, the agent comprising:
a framework orchestrator that receives commands from a controller server of a data center over the internet;
a collector that collects event information from event sources of the edge device in a runtime interval in accordance with instructions sent from the framework orchestrator;
a processor that receives the event information from the collector and reduces the event information to relevant event information in accordance with instructions received from the framework orchestrator, the relevant event information containing less information than the event information; and
a forwarder that receives the relevant event information from the processor and forwards the relevant event information over the internet to external services of the data center;
wherein the processor reduces the event information to relevant event information by performing the operations comprising:
normalizing log messages received from log message sources of the edge device by extracting parametric tokens from the log messages that describe events recorded in the log messages using corresponding regular expressions or Grok expressions for the log messages;
computing a feature vector for each of the events of the log messages;
identifying clusters of semantically similar events based on the feature vectors of the events, each cluster containing semantically similar events that correspond to an event type;
constructing a similarity graph for events of each cluster;
computing a text rank for each event of the clusters based on the similarity graph; and
determining one or more representative log messages for each cluster of events based on the text rank of the corresponding event, wherein the relevant event information comprises the representative log messages.

2. The agent of claim 1 wherein the collector collects log messages from log messages sources of the edge device.

3. The agent of claim 1 wherein the processor reduces the event information to relevant event information by performing the operations comprising:
normalizing log messages received from log message sources of the edge device by extracting parametric tokens from the log messages that describe events recorded in the log messages using corresponding regular expressions or Grok expressions for the log messages:
computing a feature vector for each of the events of the log messages:
identifying clusters of semantically similar events based on the feature vectors of the events, each cluster containing semantically similar events that correspond to an event type;
determining one or more representative log messages for each cluster of events;
constructing a probability distribution based on numbers of events in each of the clusters;
computing a log summary divergence between the probability distribution and a baseline probability distribution for the edge device; and
in response to determining the log message summary is greater than an abnormal threshold, forming the relevant event information from the representative log messages, the probability distribution, and the log summary divergence.

4. The agent of claim 1 wherein the collector collects metrics from metric sources of the edge device.

5. The agent of claim 1 wherein the processor reduces the event information to relevant event information by performing the operations comprising:
computing an aggregated metric for each metric collected in the runtime interval;
forming the relevant event information from the aggregated metrics; and
deleting metrics in the runtime interval from storage at the edge device.

6. The agent of claim 1 wherein the processor reduces the event information to relevant event information by performing the operations comprising:
synchronizing in a time a subset of the metrics collected in the runtime interval;
computing a super metric from the synchronized metrics;
computing an aggregated super metric from the super metric;
forming the relevant event information from the aggregated super metric; and
deleting metrics in the runtime interval from storage at the edge device.

7. The agent of claim 1 wherein the processor reduces the event information to relevant event information by performing the operations comprising:
for each of metrics generated by metric sources of the edge device over the runtime interval, comparing each metric value of the metric to an abnormal threshold associated with the metric; and
in response to determining the metric violates the abnormal threshold over an unacceptable time interval,
retrieving log messages of the edge device with time stamps in a backward extended time interval from a log file,
computing an aggregated metric from metrics in a threshold violation time interval, and
forming the relevant event information from the log messages and the aggregated metrics.

8. The agent of claim 1 wherein the processor reduces the event information to relevant event information by performing the operations comprising:
computing a super metric from metrics generated by metric sources of the edge device over the runtime interval;
comparing each super metric value of the super metric to an abnormal threshold associated with the super metric; and
in response to determining the super metric violates the abnormal threshold over an unacceptable time interval,
retrieving log messages of the edge device with time stamps in a backward extended time interval from a log file,
computing an aggregated metric from metrics in a threshold violation time interval, and
forming the relevant event information from the log messages and the aggregated metric.

9. A process for reducing event information that is generated by an edge device of distributed computing system and is sent to and stored in a data center, the process comprising:
collecting event information generated by event sources of the edge device in a runtime interval;
reducing the event information to relevant event information at the edge device in accordance with instructions received from a controller server of the data center, the relevant event information containing less information than the event information;
forwarding the relevant event information over the internet to external services executed at the data center; and
storing the relevant event information in a data storage device of the data center;
wherein reducing the event information to the relevant event information at the edge device comprises:
normalizing log messages received from log message sources of the edge device by extracting parametric tokens from the log messages that describe events recorded in the log messages using corresponding regular expressions or Grok expressions for the log messages;
computing a feature vector for each of the events of the log messages;
identifying clusters of semantically similar events based on the feature vectors of the events, each cluster containing semantically similar events that correspond to an event type;
constructing a similarity graph for events of each cluster;
computing a text rank for each event of the clusters based on the similarity graph; and
determining one or more representative log messages for each cluster of events based on the text rank of the corresponding event, wherein the relevant event information comprises the representative log messages.

10. The process of claim 9 wherein collecting the event information generated by the event sources of the edge device in the runtime interval comprises collecting log messages from log message sources of the edge device.

11. The process of claim 9

Assignees

• VMware LLC

Inventors

• Sharma Anil
• Brown Darren
• Algarvio Pedro
• Beard Caleb

Classifications

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06F16/313

  Selection or weighting of terms for indexing · CPC title

• G06F16/3347

  using vector based model · CPC title

• G06F18/23213

  with fixed number of clusters, e.g. K-means clustering · CPC title

• H04L41/0604

  using filtering, e.g. reduction of information by using priority, element types, position or time · CPC title