Superpixel sampling networks

Patent metadata

Field	Value
Publication number	US-10789678-B2
Application number	US-201816130871-A
Country	US
Kind code	B2
Filing date	Sep 13, 2018
Priority date	May 1, 2018
Publication date	Sep 29, 2020
Grant date	Sep 29, 2020

Abstract

A superpixel sampling network utilizes a neural network coupled to a differentiable simple linear iterative clustering component to determine pixel-superpixel associations from a set of pixel features output by the neural network. The superpixel sampling network computes updated superpixel centers and final pixel-superpixel associations over a number of iterations.

First claim

What is claimed is:

1. A superpixel sampling network comprising:
a neural network to generate a set of pixel features from an image;
a differentiable simple linear iterative clustering component coupled to receive the pixel features from the neural network;
the differentiable simple linear iterative clustering component to:
determine initial superpixel centers;
determine pixel-superpixel associations as a differentiable exponential function of distance between the set of pixel features and the initial superpixel centers;
determine updated superpixel centers; and
wherein the differentiable simple linear iterative clustering component operates for a number of iterations, each iteration utilizing the updated superpixel centers to compute the pixel-superpixel associations, the differentiable simple linear iterative clustering component outputting the image with final pixel-superpixel associations.

2. The superpixel sampling network of claim 1 , wherein the neural network and the differentiable simple linear iterative clustering component is end-to-end trainable, the differentiable simple linear iterative clustering component utilizing soft pixel-superpixel associations.

3. The superpixel sampling network of claim 1 , wherein the pixel-superpixel associations are determined by:
Q pi t =e −D(I p ,S i t-1 ) =e −∥I p −S i t-1 ∥ 2
wherein:
Q denotes a soft association matrix;
t denotes an iteration;
p denotes a pixel identifier;
i denotes a superpixel identifier;
D denotes a distance between the pixel and the superpixel center;
I denotes an image; and
S denotes a superpixel cluster center location.

4. The superpixel sampling network of claim 1 , wherein the updated superpixel centers are determined by:

S
i
t

=

1

z
i
t

⁢

∑

p
=
1

n

⁢

Q
pi
t

⁢

I
p

,

wherein:
Q denotes a soft association matrix;
t denotes an iteration;
p denotes a pixel identifier:
i denotes a superpixel identifier;
n denotes a number of pixels in the soft association matrix;
Z denotes a normalization constant;
I denotes an image; and
S denotes a superpixel cluster center location.

5. The superpixel sampling network of claim 1 , wherein the final pixel-superpixel associations are converted to spatially-connected pixel-superpixel associations.

6. The superpixel sampling network of claim 1 , wherein the differentiable simple linear iterative clustering component utilizes a reconstruction loss function.

7. The superpixel sampling network of claim 1 , wherein the differentiable simple linear iterative clustering component utilizes a compactness loss function.

8. The superpixel sampling network of claim 1 , wherein the differentiable simple linear iterative clustering component further utilizes a sum of a compactness loss function and a reconstruction loss function.

9. The superpixel sampling network of claim 1 , wherein the initial superpixel centers are determined uniformly across the image.

10. The superpixel sampling network of claim 1 , wherein the number of iterations is based on convergence of the pixel-superpixel associations of a current iteration with a previous iteration.

11. The superpixel sampling network of claim 1 , wherein the differentiable simple linear iterative clustering component further determines small superpixels, the small superpixels below a threshold size, and merges each the small superpixels with a surrounding superpixel, the differentiable simple linear iterative clustering component assigning a different cluster ID for each spatially-connected component.

12. The superpixel sampling network of claim 1 , wherein the neural network further comprises:
a first convolution layer to:
receive the image; and
convolve the image into a first convolution layer output;

a second convolution layer to convolve the first convolution layer output into a second convolution layer output;
a first pooling layer to pool the second convolution layer output into a first pooling layer output;
a third convolution layer to convolve the first pooling layer output into a third convolution layer output;
a second pooling layer to pool the third convolution layer output into a second pooling layer output;
a fourth convolution layer to convolve the second pooling layer output into a fourth convolution layer output;
a first bilinear upsampler to upsample the third convolution layer output into a first bilinear upsampler output;
a second bilinear upsampler to upsample the fourth convolution layer output into a second bilinear upsampler output; and
a final convolution layer to:
concatenate the image, the second convolution layer output, the first bilinear upsampler output, and the second bilinear upsampler output; and
generate by convolution the set of pixel features.

13. The superpixel sampling network of claim 2 , wherein the differentiable simple linear iterative clustering component further converts the soft pixel-superpixel associations to hard pixel-superpixel associations.

14. The superpixel sampling network of claim 6 , wherein the reconstruction loss function is:
L recon = ( R,R *)= ( R,{tilde over (Q)}{circumflex over (Q)} T R )
wherein:
L recon denotes the reconstruction loss function;
L denotes a task-specific loss-function;
R denotes pixel properties;
R* denotes a pixel representation;
Q ˜ denotes a row-normalized soft association matrix;
Q{circumflex over ( )} denotes a column-normalized soft association matrix; and
T denotes matrix transposition.

15. The superpixel sampling network of claim 7 , wherein the compactness loss function is:
L compact =∥I xy −Ī xy ∥ 2

wherein:
L compact denotes the compactness loss function;
I XY denotes positional pixel features; and
Ī −XY denotes an inverse mapping of positional pixel features.

16. The superpixel sampling network of claim 8 , wherein the compactness loss function is scaled by a factor of an order of [[10{circumflex over ( )}−5]]10 −5 .

17. A superpixel sampling network comprising:
a neural network to input an image and to generate a set of pixel features from the image; and
a differentiable simple linear iterative clustering component coupled to the neural network;
the differentiable simple linear iterative clustering component to determine pixel-superpixel associations from the set of pixel features and initial superpixel centers by:
Q pi t =e −D(I p ,S i t-1 ) =e −∥I p −S i t-1

Assignees

• Nvidia Corp

Inventors

• Jampani Varun
• Sun Deqing
• Liu Ming-Yu
• Kautz Jan

Classifications

• G06T3/4046 Primary

  using neural networks · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title