Fusing predictions for end-to-end panoptic segmentation

Patent metadata

Field	Value
Publication number	US-10796201-B2
Application number	US-201816125529-A
Country	US
Kind code	B2
Filing date	Sep 7, 2018
Priority date	Sep 7, 2018
Publication date	Oct 6, 2020
Grant date	Oct 6, 2020

Abstract

A method for controlling a vehicle based on a panoptic map includes receiving an input from at least one sensor of the vehicle. The method also includes generating an instance map and a semantic map from the input. The method further includes generating the panoptic map from the instance map and the semantic map based on a binary mask. The method still further includes controlling the vehicle based on the panoptic map.

First claim

What is claimed is:

1. A method for controlling a vehicle based on a panoptic map, comprising:
receiving an input from at least one sensor of the vehicle;
generating an instance map and a semantic map from the input;
generating, based on the input, a context map identifying at least one of scene depth, an edge of the objects, surface normals of the objects, or an optical flow of the objects;
generating a binary mask based on the input, the instance map, and the semantic map;
generating the panoptic map by applying the binary mask to the instance map, the context map, and the semantic map; and
controlling the vehicle based on the panoptic map.

2. The method of claim 1 , in which:
the instance map identifies each instance of a countable object; and
the semantic map associates each pixel in the input with one of a plurality of labels.

3. The method of claim 1 , further comprising generating the instance map and the semantic map with a different neural network.

4. The method of claim 1 , further comprising generating the binary mask with an artificial neural network.

5. The method of claim 4 , in which the binary mask is used to determine whether a pixel is associated with a uniquely identifiable instance of an object in the input.

6. The method of claim 4 , further comprising training the artificial neural network to generate the binary mask based on a training input labeled with object instances.

7. An apparatus for controlling a vehicle based on a panoptic map, the apparatus comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor configured:
to receive an input from at least one sensor of the vehicle;
to generate an instance map and a semantic map from the input;
to generate, based on the input, a context map identifying at least one of scene depth, an edge of the objects, surface normals of the objects, or an optical flow of the objects;
to generate a binary mask based on the input, the instance map, and the semantic map;
to generate the panoptic map by applying the binary mask to the instance map, the context map, and the semantic map; and
to control the vehicle based on the panoptic map.

8. The apparatus of claim 7 , in which:
the instance map identifies each instance of a countable object; and
the semantic map associates each pixel in the input with one of a plurality of labels.

9. The apparatus of claim 7 , in which the at least one processor is further configured to generate the instance map and the semantic map with a different neural network.

10. The apparatus of claim 7 , in which the at least one processor is further configured to generate the binary mask with an artificial neural network.

11. The apparatus of claim 10 , in which the binary mask is used to determine whether a pixel is associated with a uniquely identifiable instance of an object in the input.

12. The apparatus of claim 10 , in which the at least one processor is further configured to train the artificial neural network to generate the binary mask based on a training input labeled with object instances.

13. A non-transitory computer-readable medium having program code recorded thereon for controlling a vehicle based on a panoptic map, the program code executed by a processor and comprising:
program code to receive an input from at least one sensor of the vehicle;
program code to generate an instance map and a semantic map from the input;
program code to generate, based on the input, a context map identifying at least one of scene depth, an edge of the objects, surface normals of the objects, or an optical flow of the objects;
program code to generate a binary mask based on the input, the instance map, and the semantic map;
program code to generate the panoptic map by applying the binary mask to the instance map, the context map, and the semantic map; and
program code to control the vehicle based on the panoptic map.

14. The non-transitory computer-readable medium of claim 13 , in which:
the instance map identifies each instance of a countable object; and
the semantic map associates each pixel in the input with one of a plurality of labels.

15. The non-transitory computer-readable medium of claim 13 , in which the program code further comprises program code to generate the binary mask with an artificial neural network.

Assignees

• Toyota Res Inst Inc

Inventors

• Li Jie
• Bhargava Arjun
• Raventos Knohr Allan Ricardo
• Gaidon Adrien David

Classifications

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title