Medical scan labeling quality assurance system and methods for use therewith

Patent metadata

Field	Value
Publication number	US-11734629-B2
Application number	US-202117455066-A
Country	US
Kind code	B2
Filing date	Nov 16, 2021
Priority date	Nov 21, 2018
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

A medical scan system is operable to receive a set of labeling data corresponding to a set of medical scans from each of a set of client devices corresponding to a set of users. The set of medical scans and each set of labeling data is transmitted to an expert client device associated with an expert user, and a set of golden labeling data and a plurality of sets of correction data are received from the expert client device. A set of performance score data is generated based on the plurality of sets of correction data, and each performance score data of the set of performance score data is assigned to a corresponding one of the set of users. An updated training set that includes the set of golden labeling data is generated, and a medical scan analysis function is retrained based on the updated training set.

First claim

What is claimed is:

1. A medical scan system, comprising:
at least one processing system that includes a processor; and
at least one memory that stores executable instructions that, when executed by the processing system, cause the medical scan system to:
train a medical scan analysis function based on a training set, the medical scan function utilizing artificial intelligence;
transmit a set of medical scans to a set of client devices associated with a set of users;
receive a set of labeling data from each of the set of client devices, wherein each set of labeling data is generated by a corresponding one of the set of client devices, wherein each set of labeling data includes labeling data for each of the set of medical scans, and wherein the labeling data for each of the set of medical scans is generated by the corresponding one of the set of client devices in response to at least one prompt to provide the labeling data via a first interactive interface in conjunction with display of the each of the set of medicals scans;
transmit the set of medical scans and each set of labeling data to an expert client device associated with an expert user that is not included in the set of users;
receive a set of golden labeling data and a plurality of sets of correction data from the expert client device, wherein the set of golden labeling data is generated by the expert client device and includes golden labeling data for each of the set of medical scans based on at least one prompt to provide the golden labeling data via a second interactive interface in conjunction with display of the set of medicals scans, and wherein each set of correction data is generated by the expert client device in response to at least one additional prompt to provide the each of the set of correction data via the second interactive interface in conjunction with display of each corresponding set of labeling data; and
generate a set of performance score data by generating performance score data for each corresponding set of labeling data based on a corresponding set of correction data of the plurality of sets of correction data;
transmit each correction data of the set of correction data to a corresponding one of the set of client devices for display;
assign each performance score data of the set of performance score data to a corresponding one of the set of users that generated the corresponding set of labeling data;
generate an updated training set that includes the set of golden labeling data;
retrain the medical scan analysis function utilizing the artificial intelligence based on the updated training set, wherein model parameters of the medical scan analysis function are updated, based on the updated training set, to improve performance of the medical scan analysis function; and
generate inference data for another medical scan by performing the medical scan analysis function utilizing the artificial intelligence on image data of the another medical scan, wherein the inference data indicates an inferred abnormality.

2. The medical scan system of claim 1 , wherein the set of medical scans are transmitted to the set of client devices associated with the set of users based on elapsing of a scheduled interval.

3. The medical scan system of claim 1 , wherein the set of users and the expert user are selected from a plurality of users of the medical scan system.

4. The medical scan system of claim 1 , wherein each set of correction data includes comment data, and wherein the comment data is generated by the expert client device in response to text entered by the expert user in at least one text box displayed by the second interactive interface.

5. The medical scan system of claim 1 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
transmit each correction data of the set of correction data to all of the set of client devices for display in conjunction with the set of medical scans.

6. The medical scan system of claim 1 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
select a second set of users from a user database, wherein a set difference between the second set of users and the set of users is non-null; and
transmit the set of medical scans and the set of correction data to each one of a second set of client devices corresponding to the second set of users for display, via a third interactive interface, in conjunction with set of medical scans.

7. The medical scan system of claim 6 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
determine a common error in the sets of labeling data for at least one of the set of medical scans based on the set of golden labeling data; and
identify at least one set of labeling data that includes the common error;
wherein only the at least one of the set of labeling data that includes the common error and only the at least one of the of the set of medical scans corresponding to the common error are transmitted to the second set of client devices, and wherein the third interactive interface indicates the common error.

8. The medical scan system of claim 7 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
generate labeling commonality data by comparing the sets of labeling data to each other to determine similar ones of the sets of labeling data for at least one of the set of medical scans; and
generate common error data based on comparing the labeling commonality data to the golden labeling data to determine a proper subset of the similar ones of the sets of labeling data that compares unfavorably to the golden labeling data;
wherein the common error is determined based on the common error data.

9. The medical scan system of claim 7 , wherein the common error is identified by the expert client device in response to at least one additional prompt to provide the common error data via the second interactive interface.

10. The medical scan system of claim 1 , wherein the executable instructions, when executed by the at least one processing system, further cause the medical scan system to:
determine that a scheduled interval has elapsed since transmission of the set of medical scans to the set of client devices;
select a second set of users from a user database after the scheduled interval has elapsed since selection of the set of users in response to determining the scheduled interval has elapsed;
select a second set of medical scans from a medical scan database in response to determining the scheduled interval has elapsed, wherein an intersection between the set of medical scans and the second set of medical scans is null;
transmit the second set of medical scans, via a network, to a second set of client devices associated with the second set of users, wherein the second set of medical scans are displayed to the second set of users via the first interactive interface displayed by a second set of display devices corresponding to the second set of client devices;
receive a second set of labeling data from each of the second set of client devices via the network, wherein each second set of labeling data is generated by a corresponding one of the second set of client devices, wherein each second set of labeling data includes second labeling data for each of the second set of medical scans, and wherein the second labeling data for each of the second set of medical scans is generated by the corresponding one of the second set of client devices in response to at least one pr

Assignees

• Enlitic Inc

Inventors

• Lyman Kevin
• Upton Anthony
• Lints Lionel
• Covington Ben
• Rhodes Alexander

Classifications

• G06T12/20

  Inverse problem, i.e. transformations from projection space into object space · CPC title

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title