Scalable fine-grained resource count metrics for cloud-based data catalog service

Patent metadata

Field	Value
Publication number	US-11876875-B2
Application number	US-202117497186-A
Country	US
Kind code	B2
Filing date	Oct 8, 2021
Priority date	Oct 8, 2021
Publication date	Jan 16, 2024
Grant date	Jan 16, 2024

Abstract

In some aspects, an aggregation system of a cloud system may receive, by an aggregation system of a cloud system and from a data catalog, one or more events from one or more data sources of the cloud system, the one or more data sources having one or more resource types. The aggregation system may store event data to a message queue. The aggregation system of a cloud system may process, by one or more workers of the cloud system, a subset of events from the message queue at a time. The aggregation system may for each event in the subset of events: determine whether an event source has been updated based on information in the event. The aggregation system may in accordance with a determination that the event source has been updated, send an updated status to a dashboard, the dashboard configured to be displayed on a user device.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, by an aggregation system of a cloud system and from a data catalog, one or more events from one or more data sources of the cloud system, the one or more data sources having one or more resource types;
storing, by the aggregation system, event data to a message queue;
separating, by the aggregation system, the event data in the message queue into a set of processed events and a set of unprocessed events;
processing, by one or more workers of the cloud system, subsets of the set of unprocessed events from the event data in the message queue, where the processing of the subsets of the set of unprocessed events by the one or more workers is coordinated using a high watermark that indicates a point in the message queue separating the set of processed events from the set of unprocessed events;
for each event in each of the subsets of the set of unprocessed events:
determining whether an event source has been updated based on information in the event;

in accordance with a determination that the event source has been updated, sending an updated status to a dashboard, the dashboard configured to be displayed on a user device; and
moving, by at least one of the one or more workers of the cloud system, the high watermark in response to processing the subset of the set of unprocessed events.

2. The method of claim 1 , wherein the one or more resource types include at least one of: data assets, data entities, filename patterns, logical data entities, attributes, glossaries, categories, terms, data catalog tags, or jobs.

3. The method of claim 1 , further comprising:
receiving, by the aggregation system, an event from a data source with a new resource type without the aggregation system being reconfigured.

4. The method of claim 1 , wherein the message queue comprises a common queue for events from the one or more data sources and the one or more resource types in a common queue.

5. The method of claim 4 , wherein the one or more workers receives a subset of common queue events from the message queue.

6. The method of claim 1 , further comprising:
separating into one or more customer subsets before processing, where a customer subset comprises events associated with one tenant identifier (tenant ID).

7. The method of claim 1 , further comprising:
for each event in the subset of events:
identifying, by the data catalog the event source with a first status;
creating, by the data catalog, a second status for the event source based at least in part on the event; and
determining, by the data catalog, whether the event source has been updated based at least in part on a comparison of the first status and the second status.

8. A non-transitory computer-readable storage medium storing a set of instructions, that, when executed by one or more processors of an aggregation system computing device of a cloud system, cause the one or more processors to perform instructions comprising:
receiving from a data catalog one or more events from one or more data sources of the cloud system, the data sources having one or more resource types;
storing event data to a message queue;
separating, by the aggregation system, the event data in the message queue into a set of processed events and a set of unprocessed events;
processing, by one or more workers of the cloud system, subsets of the set of unprocessed events from event data in the message queue, where the processing of the subsets of the set of unprocessed events by the one or more workers is coordinated using a high watermark that indicates a point in the message queue separating the set of processed events from the set of unprocessed events;
for each event in each of the subset of the set of unprocessed events:
determining whether an event source has been updated based on information in the event;

in accordance with a determination that the event source has been updated, sending an updated status to a dashboard, the dashboard configured to be displayed on a user device; and
moving, by at least one of the one or more workers of the cloud system, the high watermark in response to processing the subset of the set of unprocessed events.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the one or more resource types include at least on of: data assets, data entities, filename patterns, logical data entities, attributes, glossaries, categories, terms, data catalog tags, and jobs.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the instructions further comprise:
receiving, by the aggregation system, an event from a data source with a new resource type without the aggregation system being configured.

11. The non-transitory computer-readable storage medium of claim 8 , wherein the message queue comprises a common queue for events from the one or more data sources and the one or more resource types in a common queue, wherein the one or more workers receives a subset of common queue events from the message queue.

12. The non-transitory computer-readable storage medium of claim 9 , wherein the instructions further comprise:
separating into one or more customer subsets before processing, where a customer subset comprises events associated with one tenant identifier (tenant ID).

13. The non-transitory computer-readable storage medium of claim 8 , further comprising:
for each event in the subset of events:
identifying, by the data catalog the event source with a first status;
creating, by the data catalog, a second status for the event source based at least in part on the event; and
determining, by the data catalog, whether the event source has been updated based at least in part on a comparison of the first status and the second status.

14. An aggregation system, comprising:
memory storing computer-executable instructions; and
one or more processors configured to access the memory, and execute the computer-executable instructions to at least:
receive from a data catalog one or more events from one or more data sources of a cloud system, the data sources having one or more resource types;
store event data to a message queue;
separate the event data in the message queue into a set of processed events and a set of unprocessed events;
process, by one or more workers of the cloud system, subsets of the set of unprocessed events from the event data in the message queue, where the processing of the subset of the set of unprocessed events by the one or more workers is coordinated using a high watermark that indicates a point in the message queue separating the set of processed events from the set of unprocessed events;
for each event in each of the subset of the set of unprocessed events:
determining whether an event source has been updated based on information in the event;

in accordance with a determination that the event source has been updated, send an updated status to a dashboard, the dashboard configured to be displayed on a user device; and
moving, by at least one of the one or more workers of the cloud system, the high watermark in response to processing the subset of the set of unprocessed events.

15. The aggregation system of claim 14 , wherein the one or more processors further execute the computer-executable instructions to at least:
receive, by the aggregation system, an event from a data source with a new resource type without the aggregation system being configured.

16. The aggregation system of claim 14 , wherein the message queue comprises a common queue for events from the one or more d

Assignees

• Oracle Int Corp

Inventors

• Banerjee Rick
• Dimri Rajiv

Classifications

• H04L67/566 Primary

  Grouping or aggregating service requests, e.g. for unified processing · CPC title

• H04L41/06

  Management of faults, events, alarms or notifications · CPC title

• H04L67/147

  Signalling methods or messages providing extensions to protocols defined by standardisation · CPC title

• H04L67/75

  Indicating network or usage conditions on the user display · CPC title

• H04L67/1001 Primary

  for accessing one among a plurality of replicated servers · CPC title