Training a neural network to predict superpixels using segmentation-aware affinity loss

Patent metadata

Field	Value
Publication number	US-10748036-B2
Application number	US-201816188641-A
Country	US
Kind code	B2
Filing date	Nov 13, 2018
Priority date	Nov 21, 2017
Publication date	Aug 18, 2020
Grant date	Aug 18, 2020

Abstract

Segmentation is the identification of separate objects within an image. An example is identification of a pedestrian passing in front of a car, where the pedestrian is a first object and the car is a second object. Superpixel segmentation is the identification of regions of pixels within an object that have similar properties An example is identification of pixel regions having a similar color, such as different articles of clothing worn by the pedestrian and different components of the car. A pixel affinity neural network (PAN) model is trained to generate pixel affinity maps for superpixel segmentation. The pixel affinity map defines the similarity of two points in space. In an embodiment, the pixel affinity map indicates a horizontal affinity and vertical affinity for each pixel in the image. The pixel affinity map is processed to identify the superpixels.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving input data corresponding to pixels in an image;
receiving ground-truth object segmentation data corresponding to the input data, wherein the ground-truth object segmentation data identifies objects in the image and the pixels within each object;
processing, by a loss module, the ground-truth object segmentation data and a superpixel map to compute correction data, the superpixel map corresponding to the input data and indicating regions of the pixels within each object; and
processing, by a pixel affinity neural network model, the input data using parameters to produce predicted pixel affinity values corresponding to the image, wherein the parameters are updated based on the correction data.

2. The computer-implemented method of claim 1 , wherein the predicted pixel affinity values are processed using a graph-based algorithm to produce the superpixel map.

3. The computer-implemented method of claim 1 , wherein the predicted pixel affinity values define similarity of adjacent pixels.

4. The computer-implemented method of claim 1 , converting the ground-truth object segmentation data into ground-truth pixel affinity values to compute the correction data.

5. The computer-implemented method of claim 4 , wherein the computing the correction data comprises computing a binary cross-entropy loss based on the ground-truth pixel affinity values and the predicted pixel affinity values.

6. The computer-implemented method of claim 5 , further comprising increasing a contribution to the binary cross-entropy loss for the pixels on the boundaries between the objects and within a pixel region.

7. The computer-implemented method of claim 1 , wherein the predicted pixel affinity values comprise a first set of values for a horizontal direction and a second set of values for a vertical direction.

8. The computer-implemented method of claim 7 , further comprising:
rotating the input data;
processing the rotated input data in the horizontal direction to produce rotated vertical pixel affinity values; and
rotating the rotated vertical pixel affinity values to produce the second set of values.

9. The computer-implemented method of claim 1 , wherein the correction data is associated with boundaries between the objects and within a pixel region.

10. The computer-implemented method of claim 1 , further comprising processing the input data to produce edge data, wherein the pixel affinity neural network model combines the edge data with intermediate predicted pixel affinity values to produce the predicted pixel affinity values.

11. A system, comprising:
a loss module configured to:
receive input data corresponding to pixels in an image;
receive ground-truth object segmentation data corresponding to the input data, wherein the ground-truth object segmentation data identifies objects in the image and the pixels within each object; and
process the ground-truth object segmentation data and a superpixel map to compute correction data, the superpixel map corresponding to the input data and indicating regions of the pixels within each object; and

a pixel affinity neural network model configured to process the input data using parameters to produce predicted pixel affinity values corresponding to the image, wherein the parameters are updated based on the correction data.

12. The system of claim 11 , wherein the predicted pixel affinity values are processed using a graph-based algorithm to produce the superpixel map.

13. The system of claim 11 , wherein the predicted pixel affinity values define similarity of adjacent pixels.

14. The system of claim 11 , wherein the loss module is further configured to convert the ground-truth object segmentation data into ground-truth pixel affinity values to compute the correction data.

15. The system of claim 14 , wherein the loss module is further configured to compute a binary cross-entropy loss based on the ground-truth pixel affinity values and the predicted pixel affinity values.

16. The system of claim 15 , the loss module is further configured to increase a contribution to the binary cross-entropy loss for the pixels on the boundaries between the objects and within a pixel region.

17. The system of claim 11 , wherein the predicted pixel affinity values comprise a first set of values for a horizontal direction and a second set of values for a vertical direction.

18. The system of claim 17 , wherein the pixel affinity neural network model is further configured to:
rotate the input data;
process the rotated input data in the horizontal direction to produce rotated vertical pixel affinity values; and
rotate the rotated vertical pixel affinity values to produce the second set of values.

19. The system of claim 11 , wherein the correction data is associated with boundaries between the objects and within a pixel region.

20. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
receive input data corresponding to pixels in an image;
receive ground-truth object segmentation data corresponding to the input data, wherein the ground-truth object segmentation data identifies objects in the image and the pixels within each object;
process the ground-truth object segmentation data and a superpixel map to compute correction data, the superpixel map corresponding to the input data and indicating regions of the pixels within each object; and
process the input data using parameters to produce predicted pixel affinity values corresponding to the image, wherein the parameters are updated based on the correction data.

Assignees

• Nvidia Corp

Inventors

• Tu Wei-Chih
• Liu Ming-Yu
• Jampani Varun
• Sun Deqing
• Yang Ming-Hsuan
• Kautz Jan

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/08

  Learning methods · CPC title