Computer vision system and method

Patent metadata

Field	Value
Publication number	US-10769744-B2
Application number	US-201816176801-A
Country	US
Kind code	B2
Filing date	Oct 31, 2018
Priority date	Oct 31, 2018
Publication date	Sep 8, 2020
Grant date	Sep 8, 2020

Abstract

An image processing method for segmenting an image, the method comprising:
receiving first image; producing a second image from said first image, wherein said second image is a lower resolution representation of said first image; processing said first image with a first processing stage to produce a first feature map; processing said second image with a second processing stage to produce a second feature map; and combining the first feature map with the second feature map to produce a semantic segmented image; wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; the number of layers in the first neural network being smaller than the number of layers in the second neural network.

First claim

The invention claimed is:

1. An image processing method for segmenting an image, the method comprising:
receiving first image;
producing a second image from said first image, wherein said second image is a lower resolution representation of said first image;
processing said first image with a first processing stage to produce a first feature map;
processing said second image with a second processing stage to produce a second feature map; and
combining the first feature map with the second feature map to produce a semantic segmented image;
wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; a number of layers in the first neural network being smaller than a number of layers in the second neural network.

2. An image processing method according to claim 1 , wherein the first feature map and the second feature map are combined at just one stage.

3. An image processing method according to claim 2 , wherein the first feature map and the second feature map are combined by adding.

4. An image processing method according to claim 2 , wherein the first feature map and the second feature map are combined at a fusion stage, wherein said fusion stage comprises upsampling the second feature map and adding the first feature map to the second feature map wherein adding comprises adding corresponding values of the upsampled second feature map and the first feature map.

5. An image processing method according to claim 4 , wherein upsampled second feature map is subjected to depthwise convolution prior to adding to the first feature map, wherein the depthwise convolution is performed with a dilation factor that is greater than 1.

6. An image processing method according to claim 5 , wherein upsampled second feature map has been subjected to depthwise convolution with a dilation factor that is greater than 1 and the first feature map are subjected to two dimensional convolution prior to adding.

7. An image processing method according to claim 1 , wherein separable convolution modules in the second processing stage are depthwise convolution modules.

8. An image processing method according to claim 1 , wherein separable convolution modules in the second processing stage are bottleneck architecture modules.

9. An image processing method according to claim 1 , wherein separable convolution modules in the second processing stage are bottleneck residual architecture modules.

10. An image processing method according to claim 1 , wherein the second image is subjected to standard convolution prior to be processed by said separable convolution modules.

11. An image processing method, according to claim 9 , wherein there are a plurality of residual bottleneck architecture modules.

12. An image processing method according to claim 11 , wherein a final standard convolution module is provided after the residual bottleneck architecture modules.

13. An image processing method according to claim 12 , wherein a pyramid pooling module is provided between the final standard convolution module and the residual bottleneck architecture modules.

14. An image processing method according to claim 1 , wherein the first processing stage comprises a plurality of depthwise separable convolution modules.

15. A method of training a model, said model for segmenting an image, the model comprising:
a first neural network having a plurality of layers comprising at least one separable convolution module configured to perform separable convolutions;
a second neural network with a plurality of layers, comprising at least one separable convolution module configured to perform separable convolutions; a number of layers in the first neural network being smaller than a number of layers in the second neural network,
the first neural network being configured to process an image at a first resolution and the second neural network being configured to process the same image at a lower resolution,
a feature map of the first and second neural networks being combined by addition at a single stage;
the training method comprising:
providing training data, the training data comprising images and semantic segmented information concerning said images;
training said model using said images as input and the semantic segmented information as output, wherein two stages are trained together.

16. A method according to claim 15 , further comprising increasing a number of filters during training, and reducing their number to discard filters of lower importance.

17. A method according to claim 15 , further comprising adapting the model during training to add a second output on said second processing stage, the method further comprising training using the images as input and determining loss by comparing with both the semantic segmented information at both the output and at the second output and updating weights during training by using the determined losses from both outputs.

18. An image processing system for segmenting an image, the system comprising:
an interface and a processor,
said interface having an image input and being adapted to receive a first image, said processor being adapted to;
produce a second image from said first image, wherein said second image is a lower resolution representation of said first image;
process said first image with a first processing stage to produce a first feature map;
process said second image with a second processing stage to produce a second feature map; and
combine the first feature map with the second feature map to produce a semantic segmented image;

wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; a number of layers in the first neural network being smaller than a number of layers in the second neural network.

19. A detection system for a vehicle, said detection system, comprising the image processing system of claim 18 , being adapted to receive an image and determine objects from said image by segmenting said image.

20. A non-transitory carrier medium carrying computer readable instructions adapted to cause a computer to perform the method of claim 1 .

Assignees

• Toshiba Kk

Inventors

• Poudel Karmatha Rudra Prasad
• Bonde Ujwal
• Liwicki Stephan
• Zach Christopher

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T1/0014 Primary

  Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023) · CPC title

• G06N3/045

  Combinations of networks · CPC title