Medical scan labeling quality assurance system

Patent metadata

Field	Value
Publication number	US-11211153-B2
Application number	US-201916363019-A
Country	US
Kind code	B2
Filing date	Mar 25, 2019
Priority date	Nov 21, 2018
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

A medical scan labeling quality assurance system is operable to transmit a selected set of medical scans to a set of client devices associated with an expert user and a selected set of users. The client devices display medical scans are displayed to the expert user and the set of users, and a set of labeling data generated via user input to each client device is received from each client device. A set of performance score data is generated based on comparing each set of labeling data to a set of golden labeling data that was received from the client device of the expert user. The set of performance score data is used to update user profiles of the set of users, and is transmitted to the set of client devices for display to the set of users.

First claim

What is claimed is:

1. A medical scan system, comprising:
a medical scan database that includes a plurality of medical scans;
a user database that includes a plurality of user profiles corresponding to a plurality of users of the medical scan system;
at least one processing system that includes a processor; and
at least one memory that stores executable instructions that, when executed by the processing system, cause the medical scan system to:
train a medical scan analysis function utilizing artificial intelligence based on a training set;
select a set of users from the user database in response to determining a scheduled interval has elapsed;
select a set of medical scans from the medical scan database;
transmit the set of medical scans, via a network, to a set of client devices associated with the set of users, wherein the set of medical scans are displayed to the set of users via a first interactive interface displayed by a set of display devices corresponding to the set of client devices;
receive a set of labeling data from each of the set of client devices via the network, wherein each set of labeling data is generated by a corresponding one of the set of client devices, wherein each set of labeling data includes labeling data for each of the set of medical scans, and wherein the labeling data for each of the set of medical scans is generated by the corresponding one of the set of client devices in response to at least one prompt to provide the labeling data via the first interactive interface in conjunction with display of the each of the set of medicals scans;
select an expert user from the user database, wherein the expert user is not included in the set of users;

transmit the set of medical scans, via the network, to an expert client device associated with the expert user, wherein the set of medical scans are displayed to the expert user via a second interactive interface displayed by an expert display device corresponding to the expert user;
transmit each set of labeling data to the expert client device, wherein each set of labeling data is displayed to the expert user via the second interactive interface in conjunction with display of the set of medical scans;
receive a set of golden labeling data from the expert client device via the network, wherein the set of golden labeling data is generated by the expert client device and includes golden labeling data for each of the set of medical scans, and wherein the golden labeling data for each of the set of medical scans is generated by the expert client device in response to at least one prompt to provide the golden labeling data via the second interactive interface in conjunction with display of the each of the set of medicals scans;
generate a set of performance score data by generating performance score data for each corresponding set of labeling data by comparing labeling data of each set of labeling data to the golden labeling data of the set of golden labeling data;
receive a set of correction data from each of the set of client devices, wherein each set of correction data corresponds to one set of labeling data, wherein each correction data of each set of correction data corresponds to one of the set of medical scans, wherein each set of correction data is generated by the expert client device in response to at least one additional prompt to provide the each of the set of correction data via the second interactive interface in conjunction with display of each corresponding set of labeling data, and wherein the set of performance score data is further generated based on the set of correction data; and
transmit each correction data of the set of correction data to a corresponding one of the set of client devices for display, via the first interactive interface, in conjunction with set of medical scans and the each performance score data;
assign each performance score data of the set of performance score data to a corresponding one of the set of users that generated the corresponding set of labeling data;
update each of a set of user profile entries in the user database for each corresponding one of the set of users based on the performance score data of the set of performance score data assigned to the corresponding one of the set of users;
transmit each performance score data of the set of performance score data to a corresponding one of the set of client devices for display, via the first interactive interface, to a corresponding one of the set of users to which the each performance score data is assigned;
generate an updated training set that includes the set of golden labeling data;
retrain the medical scan analysis function utilizing artificial intelligence based on the updated training set, wherein model parameters of the medical scan analysis function are updated, based on the updated training set, to improve performance of the medical scan analysis function; and
generate inference data for another medical scan by performing the medical scan analysis function utilizing artificial intelligence on image data of the another medical scan, wherein the inference data indicates an inferred abnormality.

2. The medical scan system of claim 1 , wherein each set of correction data includes comment data, and wherein the comment data is generated by the expert client device in response to text entered by the expert user in at least one text box displayed by the second interactive interface.

3. The medical scan system of claim 1 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
transmit each correction data of the set of correction data to all of the set of client devices for display in conjunction with the set of medical scans.

4. The medical scan system of claim 1 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
select a second set of users from the user database, wherein a set difference between the second set of users and the set of users is non-null; and
transmit the set of medical scans and the set of correction data to each one of a second set of client devices corresponding to the second set of users for display, via a third interactive interface, in conjunction with set of medical scans.

5. The medical scan system of claim 4 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
determine a common error in the sets of labeling data for at least one of the set of medical scans based on the set of golden labeling data; and
identify at least one set of labeling data that includes the common error;
wherein only the at least one of the set of labeling data that includes the common error and only the at least one of the of the set of medical scans corresponding to the common error are transmitted to the second set of client devices, and wherein the third interactive interface indicates the common error.

6. The medical scan system of claim 5 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
generate labeling commonality data by comparing the sets of labeling data to each other to determine similar ones of the sets of labeling data for at least one of the set of medical scans; and
generate common error data based on comparing the labeling commonality data to the golden labeling data to determine a proper subset of the similar ones of the sets of labeling data that compares unfavorably to the golden labeling data;
wherein the common error is determined based on the common error data.

7. The medical scan system of claim 5 , wherein the common error is identified

Assignees

• Enlitic Inc

Inventors

• Lyman Kevin
• Upton Anthony
• Lints Lionel
• Covington Ben
• Rhodes Alexander

Classifications

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06Q10/06315 Primary

  Needs-based resource requirements planning or analysis · CPC title