Asynchronous region-of-interest adjudication for medical images

Patent metadata

Field	Value
Publication number	US-11922668-B2
Application number	US-202117520538-A
Country	US
Kind code	B2
Filing date	Nov 5, 2021
Priority date	Nov 5, 2020
Publication date	Mar 5, 2024
Grant date	Mar 5, 2024

Abstract

A set of user interface tools is described facilitating asynchronous adjudication of one or more regions-of-interest in a medical image by a group of two or more graders, each of which has access to the set of tools in a workstation environment. The set of tools include (1) a feature for enabling the graders to assess the medical image and manually delineate one or more specific regions-of-interest (ROI) in the medical image, (2) a feature for assessing the ROI(s) delineated by other graders, including display of the ROI delineated by other graders; (3) dialog features for explaining and discussing the assessments, including a text feature for discussing the assessments. The dialog features and the ROIs delineated by all the graders are visible to all the graders on the workstation display as they collectively adjudicate the medical image in a round-robin manner. The set of tools further include (4) a feature for manually verifying grader agreement with the other graders' assessments.

First claim

We claim:

1. A method of performing an asynchronous adjudication of one or more regions-of-interest in a medical image by a group of two or more graders, each of which has access to a set of tools at a workstation in which the workstation includes a display displaying the medical image, comprising:
providing each of the graders with a first user interface feature enabling the grader to assess the medical image and manually delineate one or more specific regions-of-interest (ROI) in the medical image on the workstation display;
providing each of the graders with a second user interface feature for assessing the ROI delineated by other graders, including display of the ROI delineated by other graders;
providing each of the graders with dialog features for explaining and discussing the assessments, including a text feature permitting a grader to ask questions about a ROI delineated by other graders and permitting a grader to answer questions about the ROI they delineated wherein the dialog features and the ROIs delineated by all the graders are visible to all the graders on their respective workstation display as they adjudicate the medical image in a round-robin, asynchronous manner;
providing each of the graders with a third user interface feature for manually verifying grader agreement; and
responsive to receiving indications of grader agreement from all of the graders via respective third user interface features, concluding the adjudication of the medical image.

2. The method of claim 1 , further comprising, subsequent to providing the first user interface features to a first grader of the group of two or more graders:
receiving, from the first grader via the first user interface feature, a manual delineation of a first specific ROI; and
subsequently receiving, from a second grader of the group of two or more graders, a request to perform asynchronous adjudication of the one or more ROIs of a medical image, wherein providing each of the graders with the first user interface feature and the second user interface feature comprises, responsive to receiving the request from the second grader, providing the second grader with the first user interface feature and the second user interface feature that includes display of the first specific region-of-interest delineated by the first grader.

3. The method of claim 2 , further comprising:
receiving, from the second grader via the first user interface feature, a manually-delineation of a second specific ROI; and
subsequently receiving, from a third grader of the group of two or more graders, a request to perform asynchronous adjudication of the one or more ROIs of a medical image, wherein providing each of the graders with the first user interface feature and the second user interface feature comprises, responsive to receiving the request from the third grader, providing the third grader with the first user interface feature and the second user interface feature that includes display of the first and second specific ROIs.

4. The method of claim 1 , further comprising permitting each of the graders to generate, using the first user interface feature, a ROI in the medical image in the form of a closed curve, a point, a line, and/or a line segment.

5. The method of claim 1 , wherein the ROIs delineated by each user using the first user interface feature and any comments provided by each user using the dialog features are associated with each grader in an anonymized format.

6. The method of claim 1 , further comprising, subsequent to providing the first user interface features to a first grader of the group of two or more graders:
receiving, from a first grader of the two or more graders via the first user interface feature, a manual delineation of a first specific ROI;
subsequently providing, to a second grader of the two or more graders via the second user interface feature, a display of the first specific ROI;
subsequently receiving, from the second grader via the text feature, a first question about the first specific ROI;
responsively providing, to the first grader via the text feature, a display of the first question;
subsequently receiving, from the first grader via the text feature, a first answer; and
responsively providing, to the second grader via the text feature, a display of the first answer.

7. The method of claim 6 , wherein the tool for manually verifying grader agreement prompts the graders to verify grader agreement for the medical image as a whole.

8. The method of claim 6 , further comprising, based on ROIs generated by graders interacting with the user interface feature(s) and based on medical images related thereto, training an ML model to automatically generate an ROI for an input medical image.

9. A computing platform providing asynchronous adjudication of one or more regions-of-interest in a medical image by a group of two or more remotely located graders in communication with the computing platform over a computer network using a workstation, comprising:
a computer memory storing the medical image; and
a server distributing the medical image to the workstations used by the two or more graders, wherein the server receives data from the workstations indicating the graders' assessment of the medical image and manual delineation of one or more specific regions-of-interest (ROI) in the medical image;
wherein the server further receives data from the workstations in the form of text by which the graders assess the ROI delineated by other graders;
wherein the server further receives data from the workstations in the form of text explaining and discussing the assessments, optionally including questions about a ROI delineated by other graders and answer to questions about the ROI they delineated; wherein the server communicates with the workstations such that the said text and ROIs delineated by all the graders are visible to all the graders on the workstation display as they adjudicate the medical image in a round-robin manner;
wherein the server further receives data indicating manual verification of grader agreement and
wherein the server, responsive to receiving data indicating manual verification of grader agreement from all of the graders, concludes adjudication of the medical image.

10. The computing platform of claim 9 , wherein the server further receives, from a first grader of the group of two or more remotely located graders subsequent to receiving, from a second grader of the group of two or more remotely located graders, data indicating the second grader's delineation of a first specific ROI, a request to perform asynchronous adjudication of the one or more ROIs of a medical image, and
wherein the server communicating with the workstations such that the said text and ROIs delineated by all the graders are visible to all the graders on the workstation display as they adjudicate the medical image in a round-robin manner comprises the server, responsive to receiving the request from the first grader, communicating with a workstation of the first grader to provide a display of the first specific ROI to the first grader.

11. The computing platform of claim 9 , wherein the manually delineated ROI is in the form of a closed curve, a point, a line, and/or or a line segment.

12. The computing platform of claim 9 , wherein said text and ROIs manually delineated by the graders are presented on the workstations in an anonymized format.

13. The computing platform of claim 9 , wherein the server additionally:
communicates with a workstation of a first grader of the group of two or more remotely located graders to provide, to the first grader, a display of a first specific ROI delinea

Assignees

• Google Llc

Inventors

• Ackermann Rebecca
• Chen William
• Hughes Thad
• Daly Teagan
• Mckinney Scott
• Sayres Rory
• Duong Quang
• Stimes Jacob
• Lindley Eric
• Cruz Cristhian
• Freeman Beverly

Classifications

• G06V10/235 Primary

  based on user input or interaction · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G16H30/40

  for processing medical images, e.g. editing · CPC title

• G06V30/19147 Primary

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title