System and method for improved data consistency in data systems including dependent algorithms

Patent metadata

Field	Value
Publication number	US-10963445-B2
Application number	US-201916677394-A
Country	US
Kind code	B2
Filing date	Nov 7, 2019
Priority date	Jul 20, 2016
Publication date	Mar 30, 2021
Grant date	Mar 30, 2021

Abstract

A data system is provided for analyzing and maintaining data obtained from one or more data sources on which the data system depends. The system includes a primary database including current values used by the system and a collection of executable algorithms used to generate the data maintained in the primary database. In response to receiving a notification regarding a change in one of the data sources, a dependency database is used to establish an execution order for algorithms of the algorithm collection that are directly or indirectly dependent on the changed data. The algorithms identified in the execution order are then executed in accordance with the execution order and the corresponding result is stored in the primary database. The system may include data harvesters adapted to recognize changes in the data sources and to generate and transmit corresponding change notifications when such changes occur.

First claim

We claim:

1. A method for creating an execution plan for analyzing data in a data system comprising:
receiving a change notification, wherein the change notification includes an identifier of a changed data field of a data source;
identifying, in an algorithm metadata database, a plurality of algorithms affected by the changed data field based on the identifier of the changed data field;
determining, for each algorithm of the plurality of algorithms, a maximum depth based on dependency data for the algorithm in the algorithm metadata database and the changed data field; and
generating an execution plan including one or more layers, wherein each layer includes one or more algorithms of the plurality of algorithms, and each of the one or more algorithms in the layer has the same maximum depth.

2. The method of claim 1 , wherein the execution plan orders the one or more layers based on the maximum depth of the one or more algorithms included in each layer.

3. The method of claim 1 , further comprising:
executing the execution plan to generate an algorithm result by executing each of the one or more algorithms in each of the one or more layers.

4. The method of claim 3 , wherein each of the one or more algorithms in a layer are executed in parallel.

5. The method of claim 3 , wherein
the change notification further includes a new value or a value representative of a difference between a new value and a previous value for the changed data field, and
execution of the one or more algorithms in a root layer of the one or more layers uses the new value or the value representative of the difference.

6. The method of claim 1 , wherein one of the one or more layers is a root layer where each of the one or more algorithms included in the root layer requires a value of the changed data field.

7. A system for creating an execution plan for analyzing data in a data system comprising:
an algorithm metadata database including metadata for a plurality of executable algorithms for evaluating data; and
one or more computing devices configured to:
receive a change notification, wherein the change notification includes an identifier of a changed data field of a data source,
identify, in the algorithm metadata database, a plurality of algorithms of the plurality of executable algorithms affected by the changed data field based on the identifier of the changed data field,
determine, for each algorithm of the plurality of algorithms, a maximum depth based on dependency data for the algorithm in the algorithm metadata database and the changed data field, and
generate an execution plan including one or more layers, wherein each layer includes one or more algorithms of the plurality of algorithms, and each of the one or more algorithms in the layer has the same maximum depth.

8. The system of claim 7 , wherein the execution plan orders the one or more layers based on the maximum depth of the one or more algorithms included in each layer.

9. The system of claim 7 , wherein the one or more computing devices is further configured to execute the execution plan to generate an algorithm result by executing each of the one or more algorithms in each of the one or more layers.

10. The system of claim 9 , wherein each of the one or more algorithms in a layer are executed in parallel.

11. The system of claim 9 , wherein
the change notification further includes a new value or a value representative of a difference between a new value and a previous value for the changed data field, and
execution of the one or more algorithms in a root layer of the one or more layers uses the new value or the value representative of the difference.

12. The system of claim 7 , wherein one of the one or more layers is a root layer where each of the one or more algorithms included in the root layer requires a value of the changed data field.

13. A computing device for creating an execution plan for analyzing data in a data system comprising a non-transitory computer readable medium storing program code that, when executed by a processor of the computing device, performs the following:
receiving a change notification, wherein the change notification includes an identifier of a changed data field of a data source;
identifying, in an algorithm metadata database, a plurality of algorithms affected by the changed data field based on the identifier of the changed data field;
determining, for each algorithm of the plurality of algorithms, a maximum depth based on dependency data for the algorithm in the algorithm metadata database and the changed data field; and
generating an execution plan including one or more layers, wherein each layer includes one or more algorithms of the plurality of algorithms, and each of the one or more algorithms in the layer has the same maximum depth.

14. The computing device of claim 13 , wherein the execution plan orders the one or more layers based on the maximum depth of the one or more algorithms included in each layer.

15. The computing device of claim 13 , further comprising:
executing the execution plan to generate an algorithm result by executing each of the one or more algorithms in each of the one or more layers.

16. The computing device of claim 15 , wherein each of the one or more algorithms in a layer are executed in parallel.

17. The computing device of claim 15 , wherein
the change notification further includes a new value or a value representative of a difference between a new value and a previous value for the changed data field, and
execution of the one or more algorithms in a root layer of the one or more layers uses the new value or the value representative of the difference.

18. The computing device of claim 13 , wherein one of the one or more layers is a root layer where each of the one or more algorithms included in the root layer requires a value of the changed data field.

Assignees

• Level 3 Communications Llc

Inventors

• Gray William
• Chen Binchao
• Meyer Steven D

Classifications

• G06F16/23 Primary

  Updating · CPC title

• G06F16/2365 Primary

  Ensuring data consistency and integrity · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title