Method and system for processing an image and performing instance segmentation using affinity graphs

Patent metadata

Field	Value
Publication number	US-11881016-B2
Application number	US-201817264164-A
Country	US
Kind code	B2
Filing date	Sep 21, 2018
Priority date	Sep 21, 2018
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

A system and a method for processing an image so as to perform instance segmentation. The system/method includes: a—inputting (S1) the image (IMG) to a first neural network configured to output an affinity graph (AF), and b—inputting (S2), to a second neural network, the affinity graph and a predefined seed-map (SM), so as to determine whether other pixels belong to a same instance, and set at a first value the value of the other pixels determined as belonging to the same instance.

First claim

The invention claimed is:

1. A method for processing an image so as to perform instance segmentation, comprising:
a—inputting the image to a first neural network configured to output, for each pixel of the image, an affinity vector wherein the components of the vector are each associated with other pixels of the image at positions relative to the pixel predefined in an affinity pattern, the value of each component being set to a first value if the neural network determines that the other pixel associated with the component belongs to the same instance as the pixel of the image and set to a second value which differs from the first value if the neural network determines that the other pixel associated with the component does not belong to the same instance as the pixel of the image, the affinity vectors of all the pixels of the image forming an affinity graph,
b—inputting, to a second neural network, the affinity graph and a predefined seed-map having the resolution of the image and at least one pixel having a value set to the first value, so as to:
determine whether other pixels belong to the same instance as the at least one pixel of the seed-map having a value set to the first value, and
set at the first value the value of the other pixels determined as belonging to the same instance as the at least one pixel of the seed-map having a value set to the first value.

2. The method of claim 1 , comprising iteratively repeating step b using the seed-map modified in the previous iteration.

3. The method of claim 2 , wherein step b comprises:
for each pixel of the seed-map, determining the soft minimum vector of the affinity vector in the affinity graph associated with the pixel of the seed-map, and of a second vector having components which are the values of other pixels of the seed-map at positions relative to the pixel predefined in the affinity pattern,
determining the soft maximum of the values of said soft minimum vector,
setting the value of the pixel to said soft maximum.

4. The method of claim 1 , wherein the first neural network is a deep neural network and the second neural network is a recurrent neural network.

5. The method of claim 1 , wherein the predefined seed-map is generated by the first neural network.

6. The method of claim 1 , comprising a preliminary training step including processing a known template image and determining a loss, so as to back-propagate the loss through at least the first neural network.

7. A system for processing an image so as to perform instance segmentation, comprising:
a module for inputting the image to a first neural network configured to output, for each pixel of the image, an affinity vector wherein the components of the vector are each associated with other pixels of the image at positions relative to the pixel predefined in an affinity pattern, the value of each component being set to a first value if the neural network determines that the other pixel associated with the component belongs to the same instance as the pixel of the image and set to a second value which differs from the first value if the neural network determines that the other pixel associated with the component does not belong to the same instance as the pixel of the image, the affinity vectors of all the pixels of the image forming an affinity graph,
a module for inputting, to a second neural network, the affinity graph and a predefined seed-map having the resolution of the image and at least one pixel having a value set to the first value, so as to:
determine whether other pixels belong to the same instance as the at least one pixel of the seed-map having a value set to the first value, and
set at the first value the value of the other pixels determined as belonging to the same instance as the at least one pixel of the seed-map having a value set to the first value.

8. A non-transitory computer readable medium readable by a computer and having recorded thereon a computer program including instructions that when executed by a processor cause the processor to process an image so as to perform instance segmentation, processing the image comprising:
a—inputting the image to a first neural network configured to output, for each pixel of the image, an affinity vector wherein the components of the vector are each associated with other pixels of the image at positions relative to the pixel predefined in an affinity pattern, the value of each component being set to a first value if the neural network determines that the other pixel associated with the component belongs to the same instance as the pixel of the image and set to a second value which differs from the first value if the neural network determines that the other pixel associated with the component does not belong to the same instance as the pixel of the image, the affinity vectors of all the pixels of the image forming an affinity graph,
b—inputting, to a second neural network, the affinity graph and a predefined seed-map having the resolution of the image and at least one pixel having a value set to the first value, so as to:
determine whether other pixels belong to the same instance as the at least one pixel of the seed-map having a value set to the first value, and
set at the first value the value of the other pixels determined as belonging to the same instance as the at least one pixel of the seed-map having a value set to the first value.

Assignees

• Toyota Motor Europe
• Univ Leuven Kath
• Univ Leuven Kath

Inventors

• Shimizu Hiroaki
• De Brabandere Bert
• Neven Davy
• Proesmans Marc
• Van Gool Luc

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V20/00 Primary

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title