Learning affinity via a spatial propagation neural network

Patent metadata

Field	Value
Publication number	US-10762425-B2
Application number	US-201816134716-A
Country	US
Kind code	B2
Filing date	Sep 18, 2018
Priority date	Sep 26, 2017
Publication date	Sep 1, 2020
Grant date	Sep 1, 2020

Abstract

A spatial linear propagation network (SLPN) system learns the affinity matrix for vision tasks. An affinity matrix is a generic matrix that defines the similarity of two points in space. The SLPN system is trained for a particular computer vision task and refines an input map (i.e., affinity matrix) that indicates pixels the share a particular property (e.g., color, object, texture, shape, etc.). Inputs to the SLPN system are input data (e.g., pixel values for an image) and the input map corresponding to the input data to be propagated. The input data is processed to produce task-specific affinity values (guidance data). The task-specific affinity values are applied to values in the input map, with at least two weighted values from each column contributing to a value in the refined map data for the adjacent column.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving an input map defining properties of pixels in an image;
receiving task-specific affinity values for the pixels in the image; and
processing, by a spatial linear propagation module, the input map and the task-specific affinity values to produce refined map data, wherein at least two task-specific affinity values aligned in a first pixel dimension are applied to spatially corresponding values in the input map to generate each refined value of the refined map data.

2. The computer-implemented method of claim 1 , wherein each refined value is in a row or column adjacent to the at least two task-specific affinity values.

3. The computer-implemented method of claim 2 , wherein the processing comprises recursively applying columns of the task-specific affinity values to the spatially corresponding values in the input map such that the refined values calculated for a column contribute to the refined value in an adjacent column.

4. The computer-implemented method of claim 3 , wherein the recursively applying is performed for the columns in a first direction along the first pixel dimension, for the columns in a second direction opposing the first direction, for rows in a third direction along a second pixel dimension, and for rows in a fourth direction opposing the third direction to compute four intermediate values for each pixel of the refined data map.

5. The computer-implemented method of claim 4 , wherein the processing further comprises combining the four intermediate values for each pixel of the refined data map to produce the refined value for the pixel.

6. The computer-implemented method of claim 1 , wherein a sum of the task-specific affinity values in a column is less than or equal to one.

7. The computer-implemented method of claim 1 , wherein a guidance neural network model generates the task-specific affinity values based on input data corresponding to the input map.

8. The computer-implemented method of claim 7 , wherein the guidance neural network model is jointly trained with the spatial linear propagation module using a training dataset for the task.

9. The computer-implemented method of claim 7 , wherein the guidance neural network model is a convolutional neural network model.

10. The computer-implemented method of claim 1 , wherein the input map is coarse segmentation data for an image and the refined map data is refined segmentation data for the image.

11. The computer-implemented method of claim 1 , wherein the input map is color values associated a subset of the pixels within an image and the refined map data is colorized version of the image.

12. The computer-implemented method of claim 1 , wherein the input map is segmentation data for an image in a video sequence, the task-specific affinity values are motion affinity values, and the refined map data is segmentation data for a subsequent image in the video sequence.

13. The computer-implemented method of claim 1 , wherein the input map includes a value associated a subset of the pixels within an image and the refined map data includes a region of the image with pixels set to the value, the region including the subset of the pixels and additional pixels determined according to the task-specific affinity values.

14. A system, comprising:
a spatial linear propagation module configured to:
receive an input map defining properties of pixels in an image;
receive task-specific affinity values for the pixels in the image; and
process the input map and the task-specific affinity values to produce refined map data, wherein at least two task-specific affinity values aligned in a first pixel dimension are applied to spatially corresponding values in the input map to generate each refined value of the refined map data.

15. The system of claim 14 , wherein each refined value is in a row or column adjacent to the at least two task-specific affinity values.

16. The system of claim 15 , wherein the spatial linear propagation module is further configured to recursively apply columns of the task-specific affinity values to the spatially corresponding values in the input map such that the refined values calculated for a column contribute to the refined value in an adjacent column.

17. The system of claim 16 , wherein the spatial linear propagation module is further configured to recursively apply the columns in a first direction along the first pixel dimension, the columns in a second direction opposing the first direction, rows in a third direction along a second pixel dimension, and rows in a fourth direction opposing the third direction to compute four intermediate values for each pixel of the refined data map.

18. The system of claim 17 , wherein the spatial linear propagation module is further configured to combine the four intermediate values for each pixel of the refined data map to produce the refined value for the pixel.

19. The system of claim 14 , further comprising a guidance neural network model configured to generate the task-specific affinity values based on input data corresponding to the input map.

20. A non-transitory computer-readable media storing computer instructions for spatial linear propagation that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving an input map defining properties of pixels in an image;
receiving task-specific affinity values for the pixels in the image; and
processing, by a spatial linear propagation module, the input map and the task-specific affinity values to produce refined map data, wherein at least two task-specific affinity values aligned in a first pixel dimension are applied to spatially corresponding values in the input map to generate each refined value of the refined map data.

Assignees

• Nvidia Corp

Inventors

• Liu Sifei
• De Mello Shalini
• Gu Jinwei
• Yang Ming-Hsuan
• Kautz Jan

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2414

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06N3/09

  Supervised learning · CPC title