Adaptive Big Data Service

Patent metadata

Field	Value
Publication number	US-2020012647-A1
Application number	US-201916503186-A
Country	US
Kind code	A1
Filing date	Jul 3, 2019
Priority date	Jul 6, 2018
Publication date	Jan 9, 2020
Grant date	—

Abstract

An event processing system comprising a processor and a non-transitory computer readable medium coupled to the processor. The non-transitory computer readable medium stores computer-readable code executable to provide a domain-agnostic service for analyzing data and process heterogeneous events conforming to a plurality of schemas using the service. The domain-agnostic service comprises an adaptive data model and an adaptive processing model. The adaptive data model supports runtime event type discovery and event schema management that reacts to new schemas discovered at runtime. The adaptive processing model supports configurable partitioning of events by event type and time and is reactive to create new aggregations and analyses based on the propagation of new partitions.

First claim

What is claimed is:

1 . An event processing system comprising:
a processor; a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium storing computer-readable code, the computer-readable code executable to:
provide a domain-agnostic service for analyzing data, the domain-agnostic service comprising:
an adaptive data model that that supports runtime event type discovery and event schema management that reacts to new schemas discovered at runtime;
an adaptive processing model that supports configurable partitioning of events by event type and time and is reactive to create new aggregations and analyses based on propagation of new partitions;

process heterogeneous events conforming to a plurality of schemas using the domain-agnostic service.

2 . The event processing system of claim 1 , wherein the computer-readable code is executable to:
receive a set of heterogeneous events; and isolate events in the set of heterogeneous events to collections of homogeneous events based on event types, such that each collection of heterogenous events includes events of a corresponding event type from a plurality of event types.

3 . The event processing system of claim 2 , wherein the computer-readable code is executable to:
for a first event in the set of heterogeneous events, determine a first event type from the first event; based on a determination that the first event type is not in the adaptive data model, add the first event type to the adaptive data model; start a first collection of homogeneous events for events of the first event type and write the first event to the first collection of homogeneous events.

4 . The event processing system of claim 3 , wherein the adaptive processing model is configured to react to a propagation of a first collection of homogeneous events containing events of a first event type to:
determine a first event schema for the first event type; and store the events of the first event type from the first collection of homogeneous events to a first micro-batch according to a first structured format that reflects the first event schema.

5 . The event processing system of claim 4 , wherein the first event schema describes an event structure comprising a set of schema attributes and wherein the first structured format comprises columns corresponding to the schema attributes.

6 . The event processing system of claim 4 , wherein the adaptive processing model is configured to react to a propagation of a second collection of homogeneous events containing events of a second event type to:
determine a second event schema for the second event type; and store the events of the second event type from the second collection of homogeneous events to a second micro-batch that uses a second a structured format that reflects the second event schema, the second structured format different than the first structured format.

7 . The event processing system of claim 4 , wherein the adaptive processing model is configured to determine if the first event schema is part of the adaptive data model and based on a determination that the first event schema is not part of the adaptive data model, determine the first event schema by discovering the first event schema from the events in the first collection of homogeneous events.

8 . The event processing system of claim 4 , wherein the computer-readable code is executable to:
create a plurality of micro-batches containing events of the first event type; store the plurality of micro-batches in a first data store; select the plurality of micro-batches from the first data store based on the first event type; coalesce the plurality of micro-batches from the first data store into a first partition in a second data store.

9 . The event processing system of claim 8 , wherein adaptive processing model is configured to:
receive an aggregation rule applicable to the first event type; execute the aggregation rule to:
select, from the first data store, a first set of events that meet the aggregation rule;
select, from the second data store, a second set of events that meet the aggregation rule; and
write the first set of events and second set of events to a first aggregation.

10 . The event processing system of claim 9 , wherein the adaptive processing model is configured to react a propagation of a new micro-batch by creating a real-time aggregation of events using events from the new micro-batch and propagating the real-time aggregation of events.

11 . The event processing system of claim 9 , wherein adaptive processing model is configured to:
receive an analysis rule applicable to the first event type; execute the analysis rule to:
apply the analysis rule to the first aggregation;
store an analysis result generated by applying the analysis rule to the first aggregation.

12 . The event processing system of claim 11 , wherein applying the analysis rule to the first aggregation comprises applying a relational function or a relational transformation to events in the first aggregation.

13 . The event processing system of claim 11 , wherein the adaptive processing model is configured to react to a propagation of a real-time aggregation of events by applying the analysis rule to the real-time aggregation of events to generate a real-time analysis result.

14 . A computer program product comprising a non-transitory computer readable medium, the non-transitory computer readable medium storing computer-readable code, the computer-readable code executable to:
provide a domain-agnostic service for analyzing data, the domain-agnostic service comprising:
an adaptive data model that that supports runtime event type discovery and event schema management that reacts to new schemas discovered at runtime;
an adaptive processing model that supports configurable partitioning of events by event type and time and is reactive to create new aggregations and analyses based on propagation of new partitions;
process heterogeneous events conforming to a plurality of schemas using the domain-agnostic service.

15 . The computer program product of claim 14 , wherein the computer-readable code is executable to:
receive a set of heterogeneous events; and isolate events in the set of heterogeneous events to collections of homogeneous events based on event types, such that each collection of heterogenous events includes events of a corresponding event type from a plurality of event types.

16 . The computer program product of claim 15 , wherein the computer-readable code is executable to:
for a first event in the set of heterogeneous events, determine a first event type from the first event; based on a determination that the first event type is not in the adaptive data model, add the first event type to the adaptive data model; start a first collection of homogeneous events for events of the first event type and write the first event to the first collection of homogeneous events.

17 . The computer program product of claim 16 , wherein the computer-readable code is executable to:
determine a first event schema for the first event type; and store the events of the first event type from the first collection of homogeneous events to a first micro-batch according to a first structured format that reflects the first event schema.

18 . The computer program product of claim 17 , wherein the computer-readable code is executable to, wherein the first event schema describes an event structure comprising a set of schema attributes

Assignees

• Open Text Sa Ulc

Inventors

• Johnson Iii Claiborne Holt
• Jaskiewicz Michael Gerard

Classifications

• G06F16/213 Primary

  with details for schema evolution support · CPC title

• G06F16/24556 Primary

  Aggregation; Duplicate elimination · CPC title

• G06F16/2219

  Large Object storage; Management thereof · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title