Intelligent updating and data processing for deployed machine learning models

Patent metadata

Field	Value
Publication number	US-12333442-B2
Application number	US-202117354693-A
Country	US
Kind code	B2
Filing date	Jun 22, 2021
Priority date	Jun 22, 2021
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

Particular embodiments can update a deployed machine learning model with actual entity data depending on anomalies detected in stream data, which can be stored to a computer object, such as a journal. Various embodiments map particular subsets of a larger pool of raw input data to the particular models that need the input data and store the raw input data to computer objects so that the corresponding machine learning models can make predictions according to any suitable policy or triggering event on any of the data located in the computer objects. Such mapping allows each machine learning model to continuously make predictions based on the data it needs.

First claim

What is claimed is:

1. A computer-implemented method comprising:
deriving a trained machine learning model using a set of training data;
deploying a first copy of the trained machine learning model to a first production environment;
subsequent to the deploying the first copy of the trained machine learning model:
based on a first mapping between (a) the first copy of the trained machine learning model and (b) a plurality of data source entities, maintaining a first computer object storing a first set of stream data corresponding to the first copy of the trained machine learning model;
at least partially in response to detecting a first triggering event corresponding to the first copy of the trained machine learning model:
accessing, at the first computer object, the first set of stream data; and
feeding the first set of stream data to the first copy of the trained machine learning model; and
generating a first estimate by the first copy of the trained machine learning model using the first set of stream data as input;

storing the first estimate to a second computer object storing a first plurality of estimates generated by the first copy of the trained machine learning model;
scanning at least one of: the first computer object or the second computer object;
in response to the scanning, detecting an anomaly with at least one of: the first set of stream data or the first estimate; and
based at least in part on the detecting of the anomaly and the first mapping, updating the first copy of the trained machine learning model for first production environment by feeding at least a second set of stream data, associated with the plurality of data sources, to the first copy of the trained machine learning model.

2. The method of claim 1 , wherein:
the first computer object is a first journal data structure;
the second computer object is a second journal data structure;
the first journal data structure includes a plurality of timestamped streaming events; and
the second journal data structure includes a plurality of timestamped estimates generated by the trained machine learning model.

3. The method of claim 1 , wherein the first triggering event is at least one triggering event of a group of triggering events consisting of: a synchronous request for the first estimate, a time-based event, a data condition, and an entity change event.

4. The method of claim 1 , wherein the anomaly includes at least one of: data drift of the set of stream data, model drift of the trained machine learning model, bias of the trained machine learning model, model performance of the trained machine learning model, and latency of data flow for the set of stream data.

5. The method of claim 1 , wherein updating the first copy of the trained machine learning model comprises:
retraining the first copy of the trained machine learning model based at least in part on the detecting of the anomaly.

6. The method of claim 5 , further comprising, based at least in part on the feeding and the retraining of the first copy of the trained machine learning model, generating a second estimate via the first copy of the trained machine learning model.

7. The method of claim 1 , further comprising causing one or more notifications to be provided to a consuming device, wherein the one or more notifications include at least one notification of a group of notifications consisting of: the detected anomaly, the first estimate, the set of stream data, an indication of latency associated with the first estimate, data in the first computer object, and data in the second computer object.

8. The method of claim 1 , wherein:
the set of stream data is a first subset of stream data from a pool of streaming data;
the pool of streaming data comprises a plurality of stream data respectively received from a plurality of different data sources, and
the method further comprises:
based a first mapping between (a) the first copy of the machine learning model and (b) the first subset of stream data from the pool of streaming data, sending the first subset of stream data to the first copy of the trained machine learning model, wherein the generating of the first estimate is based at least in part on the first mapping; and
based a second mapping between (a) a second machine learning model and (b) a second subset of stream data from the pool of streaming data, sending a second set of stream data of the pool of streaming data to the second machine learning model, wherein:
the second set of stream data is different than the first set of stream data;
the second machine learning model is a second copy of the trained machine learning model, and
a second estimate of the second machine learning model is based at least in part on the mapping of the second subset of stream data.

9. The method of claim 1 , wherein the updating of the first copy of the trained machine learning model causes a generation of a second machine learning model, the method further comprising:
subsequent to the updating of the first copy of the trained machine learning model, reading at least one of: the first computer object and the second computer object to derive the first copy of the trained machine learning model before the first copy of the trained machine learning model was updated;
in response to the reading, evaluating the second machine learning model against the first copy of the trained machine learning model; and
based on the evaluating, causing the second machine learning model to be deployed or causing a notification to be rendered.

10. The method of claim 1 , further comprising:
deploying a second copy of the trained machine learning model to a second production environment;
subsequent to the deploying the second copy of the trained machine learning model:
based on a second mapping between (a) the second copy of the machine learning model and (b) a second plurality of data source entities, maintaining a third computer object storing a second set of stream data corresponding to the second copy of the trained machine learning model, wherein the second set of stream data is different than the first set of stream data;
at least partially in response to detecting a second triggering event corresponding to the second copy of the trained machine learning model:
accessing, at the third computer object, the second set of stream data;
feeding the second set of stream data to the second copy of the machine learning model; and
generating a second estimate by the second copy of the trained machine learning model using the second set of stream data as input;

storing the second estimate to a fourth computer object that records estimates generated by the second copy of the trained machine learning model;
scanning at least one of: the third computer object or the fourth computer object;
in response to the scanning, detecting an anomaly with at least one of: the set of stream data or the second estimate; and
based at least in part on the detecting of the anomaly, updating the second copy of the machine learning model for the second production environment by feeding at least one of the set of stream data or the second estimate to the second copy of the trained machine learning model.

11. Method of claim 10 , wherein the first set of stream data and the second set of stream data are subsets of stream data from a pool of streaming data associated with a plurality of production environments.

12. A system comprising:
one or more processors; and
one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, cause the one or more processors to perform-operations comprising:
deriving

Assignees

• Cerner Innovation Inc

Inventors

• Ramamurthy Uttam B
• Dellsperger David
• Finn Christopher S
• Davis Brandon
• Gritter James
• Patel Stephen
• Gandhi Maulik
• Sabaliauskas Adam

Classifications

• G06N20/00

  Machine learning · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title