System and method for real-time data warehouse

Patent metadata

Field	Value
Publication number	US-10496622-B2
Application number	US-201615169094-A
Country	US
Kind code	B2
Filing date	May 31, 2016
Priority date	Oct 9, 2015
Publication date	Dec 3, 2019
Grant date	Dec 3, 2019

Abstract

A method includes receiving, by a real-time Data Warehouse (rDW) from a first task, a first dataset and spreading the first dataset to produce a first plurality of objects, where the first plurality of objects includes a first object and a second object. The method also includes storing the first object in a first location in an rDW data repository and storing the second object in a second location in the rDW data repository.

First claim

What is claimed is:

1. A method performed by a real-time data warehouse (rDW) comprising a plurality of engines and a scheduler, the method comprising:
receiving, by the scheduler of the rDW, a first store request including a first dataset from a first task, wherein the rDW further comprises at least one on-SoC memory, at least one off-SoC memory, and the on-SoC memory locations and the off-SoC memory locations being accessible to the plurality of rDW engines, each of the plurality of rDW engines being a separate physical processor;
spreading, by a first one of the plurality of rDW engines, the first dataset to produce a first plurality of non-overlapping objects comprising a first object and a second object in a general object format, wherein the spreading is according to a rDW store/query definition (rQD) implemented in a real-time query language (rQL); and
storing, by the first rDW engine, the first object in a first memory location of the rDW and the second object in a second memory location of the rDW, each of the first memory location and the second memory location being one of the on-SoC memory locations of the rDW and one of the off-SoC memory locations of the rDW.

2. The method of claim 1 , wherein storing the first object in the first memory location of the rDW comprises storing the first object in a general data structure at the first memory location.

3. The method of claim 1 , further comprising:
receiving, by the scheduler of the rDW, a second store request from a second task that includes a second dataset, the second store request received after the first store request;
spreading, by the first rDW engine or the second rDW engine of the rDW, the second dataset to produce a second plurality of objects; and
storing, by the first rDW engine or the second rDW engine of the rDW, the second plurality of objects in memory locations of the rDW.

4. The method of claim 1 , further comprising:
receiving, by the scheduler of the rDW, a query; and
reading, by the first rDW engine or the second rDW engine of the rDW, at least a portion of the first object from the first memory location of the rDW in accordance with the query to produce a query result, and
based thereon transmitting, by the first rDW engine or the second rDW engine of the rDW, the query result to at least one task.

5. The method of claim 4 , wherein transmitting the query result to at least one task comprises multicasting the query result to multiple tasks.

6. The method of claim 1 , further comprising, after storing the first object:
determining, by the first rDW engine or a second rDW engine of the rDW, that a trigger condition of an auto-triggered query has been satisfied, and based thereon sending a task for the auto-triggered query to a scheduler of the rDW;
receiving, by the first rDW engine or the second rDW engine of the rDW, an auto-triggered query schedule for the auto-triggered query response to transmitting the task; and
performing, by the first rDW engine or the second rDW engine of the rDW, the auto-triggered query in accordance with the auto-triggered query schedule.

7. The method of claim 1 , further comprising sending a store response.

8. A method performed by a real-time data warehouse (rDW) comprising a plurality of engines and a scheduler, the method comprising:
receiving, by the scheduler of the rDW, a first query, wherein the rDW further comprises at least one on-SoC memory, at least one off-SoC memory, and the on-SoC memory locations and the off-SoC memory locations being accessible to the plurality of rDW engines, each of the plurality of rDW engines being a separate physical processor;
gathering, by a first one of the plurality of rDW engines, a first object from a first memory location of the rDW and a second object from a second memory location of the rDW in accordance with the first query to produce a first query result according to a rDW store/query definition (rQD) implemented in a real-time query language (rQL); and
transmitting, by the first rDW engine or a second one of the plurality of rDW engines, the first query result to at least one task.

9. The method of claim 8 , wherein reading the first object from the first memory location of the rDW comprises reading the first object from a general data structure at the first memory location.

10. The method of claim 8 , further comprising:
receiving, by the first rDW engine or a second rDW engine of the rDW, a second query;
reading, by the first rDW engine or the second rDW engine of the rDW, a second plurality of objects from memory locations of the rDW in accordance with the second query to produce a second query result; and
transmitting, by the rDW, the second query result.

11. The method of claim 10 , wherein the second query result is multicast to multiple tasks.

12. A method for generating system-on-chip (SoC) data and memory management programs, the method comprising:
obtaining, by a compiler executing on a processor, software definitions;
obtaining, by the compiler executing on the processor, hardware definitions;
performing, by the compiler executing on the processor, real-time Query Language (rQL) compilation on the software definitions and the hardware definitions to produce data warehouse instructions and data warehouse application structures for a real-time data warehouse (rDW) that includes one or more rDW engines, on-SoC memory including one or more on-SoC memory locations, and off-SoC memory including one or more off-SoC memory locations, both the on-SoC memory locations and the off-SoC memory locations being accessible to the one or more rDW engines, each of the one or more rDW engines being a separate physical processor;
applying, by the compiler executing on the processor, a toolchain to the data warehouse instructions to produce a binary file; and
loading, by the compiler executing on the processor, the binary file on at least a first one of the one or more rDW engines of the rDW.

13. The method of claim 12 , further comprising running, by the first rDW engine, the binary file during runtime, comprising storing a first object in memory and querying a second object from the memory.

14. The method of claim 12 , further comprising:
applying a toolchain to the data warehouse applications structures, application task codes, and data warehouse applications programming interfaces (APIs) to produce compute code; and
loading the compute code on a compute engine.

15. A method for system-on-chip (SoC) data and memory management, the method comprising:
receiving, by a first real-time data warehouse (rDW) engine of a rDW from a scheduler of the rDW, a first task schedule for a first task, the first task depending on a second task, the rDW including one or more rDW engines that include at least the first rDW engine, on-SoC memory including one or more on-SoC memory locations, and off-SoC memory including one or more off-SoC memory locations, both the on-SoC memory locations and the off-SoC memory locations being accessible to the plurality of rDW engines, each of the plurality of rDW engines being a separate physical processor; and
determining, by the first rDW engine of the rDW, whether the second task has been completed and based thereon adding the first task to a pending list as a pending task when the second task has not been completed and performing the first task when the second task has been completed.

16. The method of claim 15 , further comprising:
receiving, by the first rDW engine or a second rDW engine of the rDW from the scheduler, a second task schedule for the second task;
performing the second task af

Assignees

• Futurewei Technologies Inc

Inventors

• Chandra Alex Elisa
• Brown Mark
• Bhattacharya Debashis
• Gatherer Alan

Classifications

• G06F16/2455

  Query execution · CPC title

• G06F9/48

  Program initiating; Program switching, e.g. by interrupt · CPC title

• G06F16/25

  Integrating or interfacing systems involving database management systems · CPC title

• G06F16/245 Primary

  Query processing · CPC title

• G06F16/2219 Primary

  Large Object storage; Management thereof · CPC title