End to end network model for high resolution image segmentation

Patent metadata

Field	Value
Publication number	US-10860919-B2
Application number	US-201716339122-A
Country	US
Kind code	B2
Filing date	Sep 27, 2017
Priority date	Sep 27, 2017
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

The present disclosure provides systems and methods that leverage neural networks for high resolution image segmentation. A computing system can include a processor, a machine-learned image segmentation model comprising a semantic segmentation neural network and an edge refinement neural network, and at least one tangible, non-transitory computer readable medium that stores instructions that cause the processor to perform operations. The operations can include obtaining an image, inputting the image into the semantic segmentation neural network, receiving, as an output of the semantic segmentation neural network, a semantic segmentation mask, inputting at least a portion of the image and at least a portion of the semantic segmentation mask into the edge refinement neural network, and receiving, as an output of the edge refinement neural network, the refined semantic segmentation mask.

First claim

What is claimed is:

1. A computing system, comprising:
at least one processor;
a machine-learned image segmentation model comprising:
a semantic segmentation neural network, wherein the semantic segmentation neural network is trained to receive an image and, in response to receipt of the image, output a semantic segmentation mask; and
an edge refinement neural network, wherein the edge refinement neural network is trained to receive at least a portion of the image and at least a portion of the semantic segmentation mask and, in response to receipt of the at least a portion of the image and at least a portion of the semantic segmentation mask, output a refined semantic segmentation mask; and

at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
obtaining the image;
inputting the image into the semantic segmentation neural network;
receiving, as an output of the semantic segmentation neural network, the semantic segmentation mask, the semantic segmentation mask comprising the image partitioned into a plurality of segments, the semantic segmentation mask indicative of a boundary of an object;
inputting at least a portion of the image and at least a portion of the semantic segmentation mask into the edge refinement neural network; and
receiving, as an output of the edge refinement neural network, the refined semantic segmentation mask.

2. The computing system of claim 1 , wherein the operations further comprise:
blurring at least a portion of the image based at least in part on the refined semantic segmentation mask.

3. The computing system of claim 1 , wherein the image comprises a high resolution image; and
wherein inputting the image into the semantic segmentation neural network comprises:
down-scaling the high resolution image into a low resolution image; and
inputting the low resolution image into the semantic segmentation neural network.

4. The computing system of claim 3 , wherein receiving, as an output of the semantic segmentation neural network, the semantic segmentation mask comprises:
receiving, as an output of the semantic segmentation neural network, a low resolution semantic segmentation mask.

5. The computing system of claim 4 , wherein inputting at least a portion of the semantic segmentation mask into the edge refinement neural network further comprises:
up-scaling the low-resolution semantic segmentation mask into a high resolution semantic segmentation mask; and
inputting at least a portion of the high resolution semantic segmentation mask into the edge refinement neural network.

6. The computing system of claim 3 , wherein inputting at least a portion of the image into the edge refinement neural network comprises inputting at least a portion of the high resolution image into the edge refinement neural network.

7. The computing system of claim 1 , wherein inputting at least a portion of the image into the edge refinement neural network comprises inputting the at least a portion of the image into a first encoder network of the edge refinement neural network; and
wherein inputting at least a portion of the semantic segmentation mask into the edge refinement neural network comprises inputting the at least a portion of the semantic segmentation mask into a second encoder network of the edge refinement neural network.

8. The computing system of claim 7 , wherein inputting at least a portion of the image and at least a portion of the semantic segmentation mask into the edge refinement neural network further comprises:
receiving a first encoded output from the first encoder network;
receiving a second encoded output from the second encoder network; and
concatenating the first encoded output and the second encoded output into a concatenated encoded output.

9. The computing system of claim 8 , wherein inputting at least a portion of the image and at least a portion of the semantic segmentation mask into the edge refinement neural network further comprises:
providing the concatenated encoded output to a decoder network of the edge refinement neural network; and
wherein receiving, as an output of the edge refinement neural network, the refined semantic segmentation mask comprises receiving the refined semantic segmentation mask as an output of the decoder network.

10. The computing system of claim 1 , wherein the image segmentation model was trained from end to end based at least in part on a total loss function for the image segmentation model.

11. A computer-implemented method of generating a segmentation mask for at least a part of an image, comprising:
receiving, at a first neural network, first data derived from the image,
processing said first data, using the first neural network, to generate an output of the first neural network, wherein said output relates to a first segmentation mask, the first semantic segmentation mask comprising the image partitioned into plurality of segments, the first semantic segmentation mask indicative of a boundary of an object;
receiving, at a second neural network:
second data derived from the image, and
data derived from said output of the first neural network; and

generating data relating to a second segmentation mask, comprising processing, using the second neural network, the second data derived from the image and the data derived from said output of the first neural network, such that the second segmentation mask is refined with respect to the first segmentation mask.

12. The computer-implemented method as claimed in claim 11 , wherein the first data comprises image data at a first resolution and wherein the second data comprises image data at a second resolution which is higher than the first resolution.

13. The computer-implemented method as claimed in claim 12 , further comprising generating the first data, wherein generating the first data comprises reducing the resolution of at least a region of the image.

14. The computer-implemented method as claimed in claim 11 , wherein the first segmentation mask has a first resolution and wherein the second segmentation mask has a second resolution which is higher than the first resolution.

15. The computer-implemented method as claimed in claim 11 , wherein the first neural network and/or the second neural network comprise one or more convolutional layers.

16. The computer-implemented method as claimed in claim 11 , wherein the second neural network comprises:
at least two heads, wherein a first head is configured to receive the second data derived from the image, and a second head is configured to receive the data derived from an output of the first neural network; and
a concatenation component configured to concatenate data derived from the first and second heads.

17. The computer-implemented method as claimed in claim 16 , wherein the second neural network further comprises a decoder network configured to receive data derived from an output of the concatenation component, wherein the first head comprises a first encoder network and the second head comprises a second encoder network.

18. The computer-implemented method as claimed in claim 11 , wherein the first neural network is a semantic segmentation neural network.

19. The computer-implemented method as claimed in claim 11 , wherein the second neural network is an edge refinement neural network, and wherein the second segmentation mask comprises sharper edges compared to the fir

Assignees

• Google Llc

Inventors

• Kanazawa Noritsugu
• Knaan Yael Pritch

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/12 Primary

  Edge-based segmentation · CPC title