Analyzing data for inclusion in computer-based reasoning models

Patent metadata

Field	Value
Publication number	US-10816980-B2
Application number	US-201816220986-A
Country	US
Kind code	B2
Filing date	Dec 14, 2018
Priority date	Apr 9, 2018
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

Techniques are provided herein for creating well-balanced computer-based reasoning systems and using those to control systems. The techniques include receiving a request to determine whether to use one or more particular features, cases, etc. in a computer-based reasoning model (e.g., as cases or features are being added, or as part of pruning existing features or cases). Conviction measures (such as targeted or untargeted conviction, contribution, surprisal, etc.) are determined and inclusivity conditions are tested. The result of comparing the conviction measure can be used to determine whether to include or exclude the feature, case, etc. in the computer-based reasoning model. A controllable system may then be controlled using the computer-based reasoning model. Examples controllable systems include self-driving cars, image labeling systems, manufacturing and assembly controls, federated systems, smart voice controls, automated control of experiments, energy transfer systems, and the like.

First claim

What is claimed is:

1. A method comprising:
receiving, at a training and analysis system, a request to determine whether to include one or more particular cases in a computer-based reasoning model, wherein the training and analysis system executes on one or more computing devices, and is configured to execute training and analysis instructions;
determining, at the training and analysis system, one or more conviction scores for the one or more particular cases, wherein:
determining the one or more conviction scores for the one or more particular cases comprises determining, for each of the one or more particular cases,
a familiarity conviction score or a prediction conviction score, a familiarity conviction score is a measure of how much the one or more particular cases distort a model, and
a prediction conviction score is a measure of information required to describe a position of the one or more particular cases relative to existing cases;

determining, at the training and analysis system, whether the one or more conviction scores meet inclusivity conditions;
in response to determining that the one or more conviction scores meet the inclusivity conditions:
including the one or more particular cases in the computer-based reasoning model when the inclusivity conditions comprise an inclusion condition;
excluding the one or more particular cases in the computer-based reasoning model when the inclusivity conditions is an exclusion condition;

causing, with a control system, control of a controllable system with the computer-based reasoning model,
wherein determining one or more conviction scores for the one or more particular cases comprises determining the prediction conviction score for the one or more particular cases and determining the familiarity conviction score for the one or more particular cases;
wherein determining whether the one or more conviction scores meet the inclusivity conditions comprises determining that the one or more particular cases meet the inclusion condition when the prediction conviction score is above a first threshold and the familiarity conviction score is below a second threshold,
wherein the method is performed on one or more computing devices.

2. The method of claim 1 ,
wherein the one or more particular cases includes at least one label on at least one training image;
wherein causing control of the controllable system comprises causing control of a system that identifies elements of an image using the computer-based reasoning model by:
receiving an input image for labelling;
determining one or more labels for the input image based on the image and the computer-based reasoning model;
labelling the input image based on the one or more determined labels.

3. The method of claim 1 ,
wherein causing control of the controllable system comprises causing control of a vehicle using the computer-based reasoning model by:
receiving a current context for the vehicle, wherein the vehicle can be controlled using the computer-based reasoning model;
determining an action to take for the vehicle based on the current context for the vehicle and the computer-based reasoning model;
causing the vehicle to perform the determined action.

4. The method of claim 1 ,
wherein receiving a request to determine whether to include one or more particular cases comprises receiving a request to reduce the computer-based reasoning model to a particular size;
and the method further comprises:
determining a number of cases to include in the computer-based reasoning model to reduce the computer-based reasoning model to a particular size;
determining a subset of cases to include, that includes the number of cases, to include in the computer-based reasoning model based at least in part on the one or more conviction scores for cases in the computer-based reasoning model; and
including only the subset of cases to include in the computer-based reasoning model, and excluding cases from the computer-based reasoning model that are not in the subset of cases to include.

5. The method of claim 1 ,
wherein receiving a request to determine whether to include one or more particular cases comprises receiving a request to reduce the computer-based reasoning model to a particular size;
and the method further comprises:
determining a number of cases to exclude in the computer-based reasoning model to reduce the computer-based reasoning model by the particular size;
determining a subset of cases to exclude, that includes the number of cases, to exclude in the computer-based reasoning model based at least in part on the one or more conviction scores for cases in the computer-based reasoning model; and
excluding the subset of cases to exclude from the computer-based reasoning model.

6. The method of claim 1 , further comprising:
initially receiving the one or more particular cases as part of training for the computer-based reasoning model;
in response to determining that the one or more conviction scores meet the inclusion condition, sending an indication to a trainer associated with the training for the computer-based reasoning model to continue to train related to the one or more particular cases;
in response to determining that the one or more conviction scores meet the exclusion condition, sending the indication to the trainer associated with the training for the computer-based reasoning model that training is no longer needed related to the one or more particular cases.

7. The method of claim 1 , further comprising:
receiving a request for an action to take in a current context;
determining the action to take based on comparing the current context to contexts associated with cases in the computer-based reasoning model; and
responding to the request for the action to take with the determined action.

8. The method of claim 7 , further comprising:
receiving an indication that there was an anomaly associated with the determined action;
removing one or more cases associated with the determined action from the computer-based reasoning model.

9. The method of claim 7 , further comprising:
receiving an indication that there was an error associated with the determined action;
adding, to the computer-based reasoning model, one or more additional cases with contexts associated with the current context, wherein the one or more additional cases would cause a determination that the current context is associated with one of the one or more additional cases, and would cause determination that the current context would be associated with a different action than the determined action associated with the error.

10. The method of claim 1 , further comprising:
continuing to determine the one or more conviction scores for new cases and including or excluding those cases based on whether the one or more conviction scores meet the inclusivity conditions until a termination condition for inclusion or exclusion is met.

11. A method comprising:
receiving, at a training and analysis system, a request to determine whether to include one or more particular cases in a computer-based reasoning model, wherein the training and analysis system executes on one or more computing devices, and is configured to execute training and analysis instructions,
determining, at the training and analysis system, one or more conviction scores for the one or more particular cases, wherein:
determining the one or more conviction scores for the one or more particular cases comprises determining, for each of the one or more particular cases,
a familiarity conviction score or a prediction conviction score, a familiarity conviction score is a measure of how much the one or more particular cases distort

Assignees

• Diveplane Corp

Inventors

• Hazard Christopher James
• Fusting Christopher
• Resnick Michael

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N5/025

  Extracting rules from data · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title