Method for Labeling Images of Street Scenes

Patent metadata

Field	Value
Publication number	US-2016247290-A1
Application number	US-201514628808-A
Country	US
Kind code	A1
Filing date	Feb 23, 2015
Priority date	Feb 23, 2015
Publication date	Aug 25, 2016
Grant date	—

Abstract

A method labels an image of a street view by first extracting, for each pixel, an appearance feature for inferring a semantic label, a depth feature for inferring a depth label. Then, a column-wise labeling procedure is applied to the features to jointly determine the semantic label and the depth label for each pixel using the appearance feature and the depth feature, wherein the column-wise labeling procedure is according to a model of the street view, and wherein each column of pixels in the images includes at most four layers.

First claim

1 . A method for labeling an image of a street view, wherein the image includes a set of columns of pixels, comprising the steps of:
employing at least one processor executing computer executable instructions stored on at least one computer readable memory to facilitate performing the steps of: receiving the image having image pixels including respective two-dimensional features; receiving image data points corresponding to the image, the image data points including respective three-dimensional image features; extracting, for each pixel, an appearance feature from the image pixels; extracting, for each pixel, a depth feature from the image data points; and applying a column-wise labeling procedure to jointly determine a semantic label and a depth label for each pixel for each column of pixels from the set of columns of pixels of the image using both the appearance features and the depth features from their respective column of pixels, and wherein the column-wise labeling procedure is according to a model of the street view, wherein each column of pixels of the set of columns of pixels includes at most four ordered layers from a top layer or a fourth layer to a first layer or a bottom layer; processing, the column-wise labeling procedure subject to the at most four ordered layers to produce the labeled image, and the steps are performed in the processor.

2 . The method of claim 1 , wherein the first layer of the street view model represents drivable areas, a second layer represents dynamic objects common in the street, a third layer represents static objects, and the fourth layer represents sky, and wherein the depths in the layers are ordering from top to bottom.

3 . The method of claim 2 , wherein the drivable areas in the first layer include ground, grass, sidewalk.

4 . The method of claim 2 , wherein the dynamic objects in the second layer include vehicles, pedestrians, bicyclists, motorcyclists, and animals.

5 . The method of claim 2 , wherein the static objects in the third layer include buildings, bridges, and trees.

6 . The method of claim 1 , wherein the appearance features is determined using a neural network learned from a labeled dataset.

7 . The method of claim 1 , wherein the depth feature is determined from disparity matching cost obtained from a stereo image.

8 . The method of claim 1 , wherein the depth label for a pixel in the top layer of a column of pixels is greater than the depth label of a pixel in the remaining lower layers of the column of pixels.

9 . The method of claim 1 , wherein the depth label for a pixel in the bottom layer of a column of pixels is determined by a ground plane constraint.

10 . The method of claim 1 , wherein the image is acquired by a camera device or an image acquiring device, and the labeled image is an output on a display, such that the display and the processor are arranged in a vehicle.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Liu Ming-Yu
• Ramalingam Srikumar
• Tuzel Oncel

Classifications

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06K9/4604

  Physics · mapped topic

• G06T7/0081 Primary

  Physics · mapped topic

• G06V40/53

  Measures to keep reference information secret, e.g. cancellable biometrics · CPC title