Labeling using interactive assisted segmentation

Patent metadata

Field	Value
Publication number	US-11170264-B2
Application number	US-201916428198-A
Country	US
Kind code	B2
Filing date	May 31, 2019
Priority date	May 31, 2019
Publication date	Nov 9, 2021
Grant date	Nov 9, 2021

Abstract

Subject matter regards improving image segmentation or image annotation. A method can include receiving, through a user interface (UI), for each class label of class labels to be identified by the ML model and for a proper subset of pixels of the image data, data indicating respective pixels associated with the class label, partially training the ML model based on the received data, generating, using the partially trained ML model, pseudo-labels for each pixel of the image data for which a class label has not been received, and receiving, through the UT, a further class label that corrects a pseudo-label of the generated pseudo-labels.

First claim

What is claimed is:

1. A method of annotating image data, the method comprising:
receiving, through a user interface (UI), for each class label of at least three class labels to be identified by a machine learning (ML) model and for only a proper subset of pixels of the image data, data indicating respective pixels associated with the class label;
partially training the ML model based on the received data to generate a partially trained ML model;
generating, using the partially trained ML model, pseudo-labels for each pixel of the image data for which a class label has not been received;
display, by the UI, each pixel with a color or shading that indicates the received class label and pseudo-label, a different color or shading indicating a different class label; and
receiving, through the UI, a further class label that corrects a pseudo-label of the generated pseudo-labels and associating the further class label with the pixel.

2. The method of claim 1 , further comprising further training the partially trained ML model based on the class labels, further class label, and pseudo-labels for which neither a class label nor a further class label has been received.

3. The method of claim 2 further comprising executing the further trained ML model on further image data to classify each pixel of the further image data or training another ML model using the image data and associated class labels, further class label, and pseudo-labels.

4. The method of claim 3 , wherein the respective class labels are provided by:
receiving, through the UI and from user operation of a first software control, data indicating a selected class label of the class labels;
receiving, through the UI and from user operation of a second software control, data indicating a pixel of a rendered image of the image data to be associated with the selected class label; and
associating, in a label buffer, the received selected class label with the pixel.

5. The method of claim 4 , wherein the pixel includes a currently associated class different than the received class label, and wherein associating, in a label buffer, the received class label with the pixel includes associating the received class label with every pixel contiguous with the pixel in the image data that has an associated class equal to the currently associated class.

6. The method of claim 3 , further comprising:
processing, prior to partial training the ML model based on the received data, the image data into a segmentation feature space, mathematical transform, or representation of the image data.

7. The method of claim 3 , further comprising:
associating, prior to partially training the ML model based on the received data to generate pseudo-labels, a class label to each pixel of the image data for which a class label was not received.

8. The method of claim 7 , further comprising associating a pixel with the pseudo-label in response to determining that a generated pseudo-label for the pixel has an associated confidence score greater than a specified threshold.

9. The method of claim 8 , wherein the threshold is specified by a user, using a third software control, through the UI.

10. The method of claim 3 , further comprising providing, by the UI, a view of the image data that includes a representation of the class label and pseudo-label overlaid on the pixel image, the class label and pseudo-label provided with an opacity specified by a user, using a fourth software control, through the UI.

11. A system for annotating image data, the system comprising:
a memory including parameters of a machine learning (ML) model for pixel classification;
a user interface (UI); and
processing circuitry to:
receive, by the UI, for each class label of at least three class labels to be identified by the ML model and for a proper subset of pixels of the image data, data indicating respective pixels associated with the class label;
display, by the UI, each pixel with a color or shading that indicates the received class label;
partially train the ML model based on the received data to generate a partially trained ML model;
generate, using the partially trained ML model, pseudo-labels for each pixel of the image data for which a class label has not been received;
display, by the UI, each pixel with the color or shading that indicates the received class label and pseudo-label;
receive, through the UI, a further class label that corrects a pseudo-label of the generated pseudo-labels; and
display, by the UI each pixel with the color or shading that indicates the received class label, pseudo-label, and further class label.

12. The system of claim 11 , wherein the processing circuitry is further to train the partially trained ML model based on the class labels, further class label, and pseudo-labels for which neither a class label nor a further class label has been received.

13. The system of claim 12 , wherein the processing circuitry is further to execute the further trained ML model on further image data to classify each pixel of the further image data or training another ML model using the image data and associated class labels, further class label, and pseudo-labels.

14. The system of claim 13 , wherein the respective class labels are provided by:
receiving, through the UI and from user operation of a first software control, data indicating a selected class label of the class labels;
receiving, through the UI and from user operation of a second software control, data indicating a pixel of a rendered image of the image data to be associated with the selected class label; and
associating, in a label buffer, the received selected class label with the pixel.

15. The system of claim 14 , wherein the pixel includes a currently associated class different than the received class label, and wherein associating, in a label buffer, the received class label with the pixel includes associating the received class label with every pixel contiguous with the pixel in the image data that has an associated class equal to the currently associated class.

16. A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations for annotative image data, the operations comprising:
receiving, through a user interface (UI), for each class label of at least three class labels to be identified by a machine learning (ML) model and for only a proper subset of pixels of the image data, data indicating respective pixels associated with the class label;
partially training the ML model based on the received data to generate a partially trained ML model;
generating, using the partially trained ML model, pseudo-labels for each pixel of the image data for which a class label has not been received;
display, by the UI, each pixel with a color or shading that indicates the received class label and pseudo-label, a different color or shading indicating a different class label; and
receiving, through the UI, a further class label that corrects a pseudo-label of the generated pseudo-labels.

17. The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise:
further training the partially trained ML model based on the class labels, further class label, and pseudo-labels for which neither a class label nor a further class label has been received; and
executing the further trained ML model on further image data to classify each pixel of the further image data or training another ML model using the image data and associated class labels, further class label, and pseudo-labels.

Assignees

• Raytheon Co

Inventors

• Sallee Philip A
• Raif Stephen J
• Talamonti James

Classifications

• G06V10/40

  Extraction of image or video features · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title