Generative adversarial networks for image segmentation

Patent metadata

Field	Value
Publication number	US-11935243-B2
Application number	US-201917263813-A
Country	US
Kind code	B2
Filing date	Jun 7, 2019
Priority date	Jun 12, 2018
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A method is provided of training a generative adversarial network for performing semantic segmentation of images. The generative adversarial network includes a generator neural network and a discriminator neural network. The method includes providing an image as input to the generator neural network, receiving a predicted segmentation map for the image from the generator neural network, providing i) the image, ii) the predicted segmentation map, and iii) ground-truth label data corresponding to the image, as distinct training inputs to the discriminator neural network, determining a set of one or more outputs from the discriminator neural network in response to said training inputs, and training the generator neural network using a loss function that is a function of said set of outputs from the discriminator neural network.

First claim

The invention claimed is:

1. A method of training a generative adversarial network for performing semantic segmentation of images, wherein the generative adversarial network comprises: a generator neural network; and a discriminator neural network comprising one or more layers before a classifier, the method comprising:
providing an image as input to the generator neural network;
receiving a predicted segmentation map for the image from the generator neural network;
providing: (i) the image, (ii) the predicted segmentation map, and (iii) ground-truth label data corresponding to the image, as distinct training inputs to the discriminator neural network;
determining a set of one or more outputs from the discriminator neural network in response to said training inputs, wherein the one or more outputs comprise one or more embedding outputs taken from at least one of the layers within the discriminator neural network; and
training the generator neural network using a loss function that is a function of said set of outputs from the discriminator neural network.

2. The method of claim 1 , wherein the loss function comprises an embedding term that represents a difference between: (i) an embedding at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network, and (ii) an embedding at the predetermined layer of the discriminator neural network when the ground-truth label data is input to the discriminator neural network.

3. The method of claim 2 , wherein the difference is an L2 distance.

4. The method of claim 2 , wherein the predetermined layer is located immediately after the first dense block that contains one or more shared convolution layers with both the image and the predicted segmentation map.

5. The method of claim 2 , wherein the predetermined layer is located immediately after the final dense block before the classifier of the discriminator neural network.

6. The method of claim 2 , wherein the loss function additionally comprises a pixel-level fitness term, and wherein the loss function comprises a weighting parameter, λ, for weighting the embedding term relative to the fitness term.

7. The method of claim 1 , further comprising training the discriminator neural network to minimize a loss on the discrimination of the discriminator neural network between predicted segmentation maps and ground-truth label data.

8. The method of claim 1 , further comprising using the trained generative adversarial network to segment a received image of a road and identify one or more lane markings in the image.

9. A computer processing system implementing a generative adversarial network for performing semantic segmentation of images, wherein the generative adversarial network comprises: a generator neural network; a discriminator neural network comprising one or more layers before a classifier; and training logic, and wherein the training logic is configured to:
provide an image as input to the generator neural network;
receive a predicted segmentation map for the image from the generator neural network;
provide: (i) the image, (ii) the predicted segmentation map, and (iii) ground-truth label data corresponding to the image, as distinct training inputs to the discriminator neural network;
determine a set of one or more outputs from the discriminator neural network in response to said training inputs, wherein the one or more outputs comprise one or more embedding outputs taken from at least one of the layers within the discriminator neural network; and
train the generator neural network using a loss function that is a function of said set of outputs from the discriminator neural network.

10. The computer processing system of claim 9 , wherein the loss function comprises an embedding term that represents a difference between: (i) an embedding at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network, and (ii) an embedding at the predetermined layer of the discriminator neural network when the ground-truth label data is input to the discriminator neural network.

11. The computer processing system of claim 10 , wherein the predetermined layer is located immediately after the first dense block that contains one or more shared convolution layers with both the image and the predicted segmentation map.

12. The computer processing system of claim 10 , wherein the predetermined layer is located immediately after the final dense block before the classifier of the discriminator neural network.

13. The computer processing system of claim 10 , wherein the loss function additionally comprises a pixel-level fitness term, and wherein the loss function comprises a weighting parameter, λ, for weighting the embedding term relative to the fitness term.

14. The computer processing system of claim 9 , wherein the training logic is further configured to train the discriminator neural network to minimize a loss on the discrimination of the discriminator neural network between predicted segmentation maps and ground-truth label data.

15. The computer processing system of claim 9 , wherein the generative adversarial network is configured, after being trained, to segment a received image of a road and identify one or more lane markings in the image.

16. A non-transitory computer readable storage medium storing instructions that, when executed by a computer processing system, cause the computer processing system to perform method of training a generative adversarial network for performing semantic segmentation of images, wherein the generative adversarial network comprises: a generator neural network; and a discriminator neural network comprising one or more layers before a classifier, the method comprising:
providing an image as input to the generator neural network;
receiving a predicted segmentation map for the image from the generator neural network;
providing: (i) the image, (ii) the predicted segmentation map, and (iii) ground-truth label data corresponding to the image, as distinct training inputs to the discriminator neural network;
determining a set of one or more outputs from the discriminator neural network in response to said training inputs, wherein the one or more outputs comprise one or more embedding outputs taken from at least one of the layers within the discriminator neural network; and
training the generator neural network using a loss function that is a function of said set of outputs from the discriminator neural network.

17. The non-transitory computer readable storage medium of claim 16 , wherein the loss function comprises an embedding term that represents a difference between: (i) an embedding at a predetermined layer of the discriminator neural network when the predicted segmentation map is input to the discriminator neural network, and (ii) an embedding at the predetermined layer of the discriminator neural network when the ground-truth label data is input to the discriminator neural network.

18. The non-transitory computer readable storage medium of claim 17 , wherein the predetermined layer is located immediately after the first dense block that contains one or more shared convolution layers with both the image and the predicted segmentation map.

19. The non-transitory computer readable storage medium of claim 17 , wherein the predetermined layer is located immediately after the final dense block before the classifier of the discriminator neural network.

20. The non-tr

Assignees

• Tomtom Global Content Bv

Inventors

• Hofmann Michael
• Baka Nora
• Nugteren Cedric
• Ghafoorian Mohsen
• Booij Olaf

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title