Method for labeling images of street scenes

Patent metadata

Field	Value
Publication number	US-9811756-B2
Application number	US-201514628808-A
Country	US
Kind code	B2
Filing date	Feb 23, 2015
Priority date	Feb 23, 2015
Publication date	Nov 7, 2017
Grant date	Nov 7, 2017

Abstract

A method labels an image of a street view by first extracting, for each pixel, an appearance feature for inferring a semantic label, a depth feature for inferring a depth label. Then, a column-wise labeling procedure is applied to the features to jointly determine the semantic label and the depth label for each pixel using the appearance feature and the depth feature, wherein the column-wise labeling procedure is according to a model of the street view, and wherein each column of pixels in the images includes at most four layers.

First claim

We claim:

1. A method for labeling an image of a street view, wherein the image includes a set of columns of pixels, comprising the steps of:
employing at least one processor executing computer executable instructions stored on at least one computer readable memory to facilitate performing the steps of:
receiving the image having image pixels including respective two-dimensional features;
receiving image data points corresponding to the image, the image data points including respective three-dimensional image features;
extracting, for each pixel, an appearance feature from the image pixels, wherein the appearance features are determined using a deep neural network learned from a labeled dataset;
extracting, for each pixel, a depth feature from the image data points; and
applying a column-wise labeling procedure to jointly determine a semantic label and a depth label for each pixel for each column of pixels from the set of columns of pixels of the image using both the appearance features and the depth features from their respective column of pixels, and wherein the column-wise labeling procedure is according to a model of the street view, wherein each column of pixels of the set of columns of pixels includes at most four ordered layers from a top layer or a fourth layer to a first layer or a bottom layer, such that the at most four ordered layers are obtained using an inference procedure that jointly estimates the sematic labels and the depth labels for each image column;
processing, the column-wise labeling procedure subject to the at most four ordered layers to produce the labeled image, and the steps are performed in the processor.

2. The method of claim 1 , wherein the first layer of the street view model represents drivable areas, a second layer represents dynamic objects common in the street, a third layer represents static objects, and the fourth layer represents sky, and wherein the depths in the layers are ordering from top to bottom.

3. The method of claim 2 , wherein the drivable areas in the first layer include ground, grass, sidewalk.

4. The method of claim 2 , wherein the dynamic objects in the second layer include vehicles, pedestrians, bicyclists, motorcyclists, and animals.

5. The method of claim 2 , wherein the static objects in the third layer include buildings, bridges, and trees.

6. The method of claim 1 , wherein the appearance features are determined using the deep neural network learned from the labeled dataset, such that the deep neural network is used to extract image features for sematic classes from the images.

7. The method of claim 1 , wherein the depth feature is determined from disparity matching cost obtained from a stereo image.

8. The method of claim 1 , wherein the depth label for a pixel in the top layer of a column of pixels is greater than the depth label of a pixel in the remaining lower layers of the column of pixels.

9. The method of claim 1 , wherein the depth label for a pixel in the bottom layer of a column of pixels is determined by a ground plane constraint.

10. The method of claim 1 , wherein the image is acquired by a camera device or an image acquiring device, and the labeled image is an output on a display, such that the display and the processor are arranged in a vehicle.

11. A method for labeling an image of a street view, wherein the image includes a set of columns of pixels, comprising the steps of:
employing at least one processor executing computer executable instructions stored on at least one computer readable memory to facilitate performing the steps of:
receiving the image having image pixels including respective two-dimensional features;
receiving image data points corresponding to the image, the image data points including respective three-dimensional image features;
extracting, for each pixel, an appearance feature from the image pixels, wherein the appearance features are determined using a deep multi-scale convolutional network from a labeled dataset used to extract image features for sematic classes from the images;
extracting, for each pixel, a depth feature from the image data points; and
applying a column-wise labeling procedure to jointly determine a semantic label and a depth label for each pixel for each column of pixels from the set of columns of pixels of the image using both the appearance features and the depth features from their respective column of pixels, and wherein the column-wise labeling procedure is according to a model of the street view, wherein each column of pixels of the set of columns of pixels includes at most four ordered layers from a top layer or a fourth layer to a first layer or a bottom layer, such that the at most four ordered layers are obtained using an inference procedure that jointly estimates the sematic labels and depth labels for each image column;
processing, the column-wise labeling procedure subject to the at most four ordered layers to produce the labeled image, and the steps are performed in the processor.

12. The method of claim 1 , wherein the inference procedure is based on a dynamic algorithm, that includes variables, h i1 , h i2 , h i3 , and h i4 , to denote y coordinates of top pixels of the at most four ordered layers, such that l i1 , l i2 , l i3 , and l i4 are semantic labels for the at most four ordered layers, and d i1 , d i2 , d i3 , and d i4 are depths of the at most four ordered layers, wherein only unknown variables are determined, for example, if the unknown variables included h 1 , h 2 , h 3 , l 2 , d 3 , then the unknown variables are determined by

R
i

⁡

(

h
2

,

h
3

,

d
3

)

≡

∑

y
=

h
4

h
3

-
1

⁢

E
A

⁡

(

i
,
y
,
??

)

+

E
D

⁡

(

i
,
y
,
∞

)

+

∑

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Liu Ming-Yu
• Ramalingam Srikumar
• Tuzel Oncel

Classifications

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06K9/4604 Primary

  Physics · mapped topic

• G06K2009/00953

  Physics · mapped topic

• G06K9/00791

  Physics · mapped topic

• G06V40/53

  Measures to keep reference information secret, e.g. cancellable biometrics · CPC title