Assessing unreliability of clinical risk prediction

Patent metadata

Field	Value
Publication number	US-11551817-B2
Application number	US-202016741991-A
Country	US
Kind code	B2
Filing date	Jan 14, 2020
Priority date	Jan 14, 2020
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

Aspects of the invention include includes identifying a respective estimated clinical risk score for each of a first group of patients and a second group of patients. An alternative probability estimate is generated using a same set of inputs used to determine each respective estimated clinical risk score. An unreliability of a patient's clinical risk score is determined based at least in part on a feature of the patient and on a difference between the alternative probability estimate and the determined respective estimated clinical risk score.

First claim

What is claimed is:

1. A computer-implemented method comprising:
using a neural network as a trained risk model for identifying a respective estimated clinical risk score for each of a first group of patients having an adverse outcome within a specified time to a treatment and a second group of the patients having not experienced the adverse outcome within the specified time to the treatment, the respective estimated clinical risk score for the first group of the patients and the second group of the patients having been determined by the trained risk model using a set of inputs as original data, there being an imbalance in a number of the patients in the first group and the second group;
training a generative model using the first group of the patients having the adverse outcome as training data within the specified time to the treatment, wherein the generative model learns an underlying data distribution in order to generate synthetic first group data, wherein the generative model is a multivariate normal probability density function, wherein the training data of the generative model comprises a portion used to train the neural network;
generating synthetic first group data using the generative model to alleviate the imbalance, the synthetic first group data generated by the generative model is based at least in part on the first group of the patients having the adverse outcome within the specified time to the treatment, the generative model having been trained using the first group of the patients, the generative model comprising machine learning algorithms, wherein the synthetic first group data augments the number of the patients in the first group;
generating an alternative probability estimate using the set of inputs used to determine each respective estimated clinical risk score, wherein the alternative probability estimate is based at least in part on the original data from electronic medical records of the patients of the first group and the second group in combination with the synthetic first group data generated by the generative model;
determining an unreliability of a patient's clinical risk score based at least in part on a feature of the patient and on a difference between the alternative probability estimate and the determined respective estimated clinical risk score; and
responsive to a determination of unreliability associated with the patient, alerting a clinician via a user interface of the unreliability associated with the patient in order to result in a modification in medication for the patient, the alerting causing an avoidance of an invasive procedure and its associated risk for the patient.

2. The computer-implemented method of claim 1 , further comprising:
generating synthetic second group data using a generative model, wherein the alternative probability estimate is based at least in part on original data from electronic medical records of the patients of the first group and the second group and the synthetic data.

3. A computer-implemented method of claim 1 , wherein a class imbalance skews in favor of the second group.

4. The computer-implemented method of claim 1 , wherein each patient from the first group shares a feature, and wherein each patient from the second group shares a feature.

5. The computer-implemented method of claim 1 , further comprising:
displaying the unreliability of the clinical risk score on a user interface,
wherein the user interface is modified from an original format to provide a visual alert of an unreliability of the clinical risk score.

6. A system comprising:
a memory having computer readable instructions; and
one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
using a neural network as a trained risk model for identifying a respective estimated clinical risk score for each of a first group of patients having an adverse outcome within a specified time to a treatment and a second group of the patients having not experienced the adverse outcome within the specified time to the treatment, the respective estimated clinical risk score for the first group of the patients and the second group of the patients having been determined by the trained risk model using a set of inputs as original data, there being an imbalance in a number of the patients in the first group and the second group;
training a generative model using the first group of the patients having the adverse outcome as training data within the specified time to the treatment, wherein the generative model learns an underlying data distribution in order to generate synthetic first group data, wherein the generative model is a multivariate normal probability density function, wherein the training data of the generative model comprises a portion used to train the neural network;
generating the synthetic first group data using the generative model to alleviate the imbalance, the synthetic first group data generated by the generative model is based at least in part on the first group of the patients having the adverse outcome within the specified time to the treatment, the generative model having been trained using the first group of the patients, the generative model comprising machine learning algorithms, wherein the synthetic first group data augments the number of the patients in the first group;
generating an alternative probability estimate using the set of inputs used to determine each respective estimated clinical risk score, wherein the alternative probability estimate is based at least in part on the original data from electronic medical records of the patients of the first group and the second group in combination with the synthetic first group data generated by the generative model;
determining an unreliability of a patient's clinical risk score based on a difference between the alternative probability estimate and the determined respective estimated clinical risk score; and
responsive to a determination of unreliability associated with the patient, alerting a clinician via a user interface of the unreliability associated with the patient in order to result in a modification in medication for the patient, the alerting causing an avoidance of an invasive procedure and its associated risk for the patient.

7. The system of claim 6 , the operations further comprising:
generating synthetic second group data a generative model, wherein the alternative probability estimate is based at least in part on original data from electronic medical records of the patients of the first group and the second group and the synthetic data.

8. The system of claim 6 , wherein a class imbalance skews in favor of the second group.

9. The system of claim 6 , wherein the first group comprises patients that have a positive outcome within a specified time to a treatment, the positive outcome being associated with the adverse outcome, and wherein the second group comprises patients that have a negative outcome with the specified time to the treatment.

10. The system of claim 6 , wherein each patient from the first group shares a feature, and wherein each patient from the second group shares a feature.

11. The system of claim 6 , the operations further comprising:
displaying the reliability of the clinical risk score on a user interface,
wherein the user interface is modified from an original format to provide a visual alert of an unreliability of the clinical risk score.

12. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to

Assignees

• IBM
• Massachusetts Inst Technology

Inventors

• Myers Paul D
• Kartoun Uri
• Severson Kristen
• Dai Wangzhi
• Ng Kenney
• Stultz Collin M

Classifications

• G16H50/30 Primary

  for calculating health indices; for individual health risk assessment · CPC title

• G16H50/20

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title

• G16H20/00

  ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06F18/24155

  Bayesian classification · CPC title