Large scale real-time multistaged analytic system using data contracts

Patent metadata

Field	Value
Publication number	US-9367619-B2
Application number	US-201113194084-A
Country	US
Kind code	B2
Filing date	Jul 29, 2011
Priority date	Jul 29, 2011
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

An analytic system may have a number of processing stages. One or more data sources may provide data to a first processing stage. The first processing stage may specify one or more data contracts having a schema describing a layout and types of data provided by the one or more data sources. Each of the processing stages may specify a respective data contract having a schema such that the processing stages may understand a layout and types of data provided as input to the processing stages. The data contracts me further specify a valid range of values for various items of data described by schemas. Data not conforming to a data contract may be automatically filtered out such that a corresponding processing stage may not be provided with the non-conforming data.

First claim

We claim as our invention:

1. A system comprising:
a receiver configured to receive heterogeneous input data that is provided by at least one of a plurality of data sources;
a memory configured to store at least one data contract that specifies a layout of the heterogeneous input data;
a processor configured to:
process, according to the layout specified in each of the at least one data contract, the heterogeneous input data without transforming the heterogeneous input data to conform to a generic schema;
filter, according to the at least one data contract, the heterogeneous input data that does not conform to the at least one data contract;
implement a collection processing stage configured to collect data provided by the at least one of a plurality of data sources;
implement an aggregation processing stage configured to process and aggregate data provided by the collection processing stage; and
implement a visualization processing stage configured to render data provided from the aggregation processing stage to an output device.

2. The system of claim 1 , where the specified layout of each of the at least one data contract comprises a schema that describes the heterogeneous input data that is provided by the corresponding one of the plurality of data sources.

3. The system of claim 1 configured to provide, via the visualization processing stage, user interface building blocks.

4. A machine-readable storage medium storing instructions for more than a transitory time period that, based on execution by at least one processor of at least one computing device that includes memory, configure the at least one computing device to:
execute, by the at least one computing device, at least one of a plurality of processing stages, where each processing stage of the plurality of processing stages includes at least one data contract that specifies a layout of heterogeneous input data received by the each processing stage that is provided by at least one of a plurality of data sources that includes the plurality of processing stages, where a first of the processing stages is a collection processing stage configured to collect data provided by the at least one of a plurality of data sources, where a second of the processing stages is an aggregation processing stage configured to process and aggregate data provided from the collection processing stage, and where a third of the processing stages is a visualization processing stage configured to render data provided from the aggregation processing stage to an output device;
process, via the each processing stage according to the layout specified in each of the corresponding at least one data contract, the heterogeneous input data received by the each processing stage without transforming the received heterogeneous input data to conform to a generic schema; and
filter, via the each processing stage according to the corresponding at least one data contract, the heterogeneous input data received by the each processing stage that does not conform to the corresponding at least one data contract.

5. The machine-readable storage medium of claim 4 , where the specified layout of each of the at least one data contract comprises a schema that describes the heterogeneous input data that is provided by the corresponding one of the plurality of data sources.

6. The machine-readable medium of claim 4 , where the instructions further configure the at least one computing device to provide, via the visualization processing stage, user interface building blocks.

7. A method performed on at least one computing device that includes at least one processor and memory, the method comprising:
executing, by the at least one computing device, at least one of a plurality of processing stages, where each processing stage of the plurality of processing stages includes at least one data contract that specifies a layout of heterogeneous input data received by the each processing stage that is provided by at least one of a plurality of data sources that includes the plurality of processing stages, where a first of the processing stages is a collection processing stage configured to collect data provided by the at least one of a plurality of data sources, where a second of the processing stages is an aggregation processing stage configured to process and aggregate data provided by the collection processing stage, and where a third of the processing stages is a visualization processing stage configured to render data provided from the aggregation processing stage, the method further comprising using, by the visualization processing to an output device;
processing, via the each processing stage according to the layout specified in each of the corresponding at least one data contract, the heterogeneous input data received by the each processing stage without transforming the received heterogeneous input data to conform to a generic schema; and
filtering, via the each processing stage according to the corresponding at least one data contract, the heterogeneous input data received by the each processing stage that does not conform to the corresponding at least one data contract.

8. The method of claim 7 , where the specified layout of each of the at least one data contract comprises a schema that describes the heterogeneous input data that is provided by the corresponding one of the plurality of data sources.

9. The method of claim 7 further comprising providing, via the visualization processing stage, user interface building blocks.

Assignees

• Katiyar Atul
• Amradkar Purushottam
• Gupta Nitin
• Microsoft Technology Licensing Llc

Inventors

• Katiyar Atul
• Amradkar Purushottam
• Gupta Nitin

Classifications

• G06F17/30864 Primary

  Physics · mapped topic

• G06F17/30846

  Physics · mapped topic

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/74

  Browsing; Visualisation therefor (end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic programme guides, H04N21/472) · CPC title