Method for semantically labeling an image of a scene using recursive context propagation

Patent metadata

Field	Value
Publication number	US-9558268-B2
Application number	US-201414463806-A
Country	US
Kind code	B2
Filing date	Aug 20, 2014
Priority date	Aug 20, 2014
Publication date	Jan 31, 2017
Grant date	Jan 31, 2017

Abstract

A method semantically labels an image acquired of a scene by first obtaining a local semantic feature for each local region in the image. The local semantic features are combined recursively to form intermediate segments until a semantic feature for the entire image is obtained. Then, the semantic feature for the entire image is decombined recursively into intermediate segments until an enhanced semantic feature for each local region is obtained. Then, each local region is labeled according to the enhanced semantic feature.

First claim

We claim:

1. A method for semantically labeling an image acquired of a scene, comprising steps of:
obtaining a local semantic feature from each local region in the image;
combining recursively the local semantic features to form intermediate segments until a semantic feature for the entire image is obtained;
decombining recursively the semantic feature for the entire image into intermediate segments until an enhanced semantic feature for each local region is obtained; and
labeling each local region according to the enhanced semantic feature, wherein the steps are performed in a processor.

2. The method of claim 1 , wherein the local regions are superpixels.

3. The method of claim 1 , wherein the local regions include one or more pixels.

4. The method of claim 1 , wherein the local features are extracted from local regions according to a multi scale convolutional neural network.

5. The method of claim 1 , wherein the combining and the decombining are according to a parse tree of the image.

6. The method of claim 1 , further comprising:
synthesizing randomly balanced binary parse trees of nodes for the combining and the decombining.

7. The method of claim 1 ,
where the mapping is according to x i =F sem (v i ;θ sem ),

wherein x i is the semantic feature, v i is the local feature, and θ sem is a semantic mapping parameter,
where the combing is according to x i,j =F sem ([x i ,x j ];θ com ),

wherein x i,j a semantic feature of a parent node obtained from child nodes x i ,x j in a binary parse tree, and θ com is a combining parameter;
where the decombining is according to {tilde over (x)} i =F dec ([{tilde over (x)} i,j ,x i ];θ dec ),

wherein {tilde over (x)} is an enhanced semantic feature of a child node obtained from a parent node {tilde over (x)} i,j and x i , and θ sem is a decombining parameter, and
wherein the labeling is according to y i =F lab ({tilde over (x)} i ;θ lab ),

wherein y i is a label, and θ sem is a labeling parameter.

8. The method of claim 6 , and further comprising:
adding side information to the nodes.

9. The method of claim 8 , wherein the side information encodes static knowledge about the nodes.

10. The method of claim 8 , wherein the side information is an average of locations of the nodes and sizes of the nodes.

11. The method of claim 7 , wherein the extracting, combining, decombining and labeling functions use neural network.

12. The method of claim 1 , wherein the local semantic features are pixel color, gradient and texture features.

13. The method of claim 7 , further comprising:
learning the parameters of the extracting, the combining, the decombining and the labeling using training data.

14. The method of claim 13 , wherein the learning minimizes a difference between true labels and predicted labels of the pixels in the training data.

15. The method of claim 7 , wherein function of the combining and the decombining are recursive.

16. The method of claim 5 , wherein the parse tree is a hierarchical segmentation representation of the image of the scene.

17. The method of claim 5 , wherein the parse tree is a hierarchical segmentation representation of a part of the image of the scene.

18. The method of claim 1 , wherein the semantic feature for the entire image is used to classify the image according to scene categories.

19. The method of claim 1 , wherein the semantic feature for the entire image is used for clustering collections of images.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Tuzel Oncel
• Liu Ming-Yu
• Sharma Abhishek

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• G06F2218/08

  Feature extraction · CPC title

• G06F18/24323

  Tree-organised classifiers · CPC title

• G06F18/24

  Classification techniques · CPC title