Method and system for generating a bird's eye view bounding box associated with an object

Patent metadata

Field	Value
Publication number	US-11527084-B2
Application number	US-202016926096-A
Country	US
Kind code	B2
Filing date	Jul 10, 2020
Priority date	Jul 10, 2020
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

A system and method for generating a bounding box for an object in proximity to a vehicle are disclosed. The method includes: receiving a three-dimensional (3D) point cloud representative of an environment; receiving a two-dimensional (2D) image of the environment; processing the 3D point cloud to identify an object cluster of 3D data points for a 3D object in the 3D point cloud; processing the 2D image to detect a 2D object in the 2D image and generate information regarding the 2D object from the 2D image; and when the 3D object and the 2D object correspond to the same object in the environment: generating a bird's eye view (BEV) bounding box for the object based on the object cluster of 3D data points and the information from the 2D image.

First claim

The invention claimed is:

1. A processor-implemented method for generating a bounding box for an object in proximity to a vehicle, the method comprising:
receiving a three-dimensional (3D) point cloud representative of an environment;
receiving a two-dimensional (2D) image of the environment;
processing the 3D point cloud to identify an object cluster of 3D data points for a 3D object in the 3D point cloud;
processing the 2D image to detect a 2D object in the 2D image and generate information regarding the 2D object from the 2D image, wherein the information from the 2D object comprises an image heading h image of the object, and an image heading uncertainty σ image 2 associated with the image heading h image of the object; and
when the 3D object and the 2D object correspond to the same object in the environment:
generating a bird's eye view (BEV) bounding box for the object based on the object cluster of 3D data points and the information from the 2D image, the generating including:
mapping the object cluster of 3D data points to a cluster of 2D data points on a 2D plane in a bird's eye view (BEV) and in a vehicle coordinate system of the vehicle;
determining and storing a group of BEV polygon points on the 2D plane in the BEV, wherein the group of BEV polygon points forms a convex hull enclosing the cluster of 2D data points on the 2D plane;
determining a center p center of the cluster of 2D data points on the 2D plane;
determining an estimated heading h obj of the object;
rotating the cluster of 2D data points on the 2D plane around the center p center based on the estimated heading h obj ;
determining a plurality of selected polygon points from the group of BEV polygon points;
determining a plurality of candidate bounding boxes, wherein each candidate bounding box is determined based on a respective selected polygon point from the plurality of selected polygon points, the determining including, for each respective polygon point of the plurality of selected polygon points:
generating four rectangle boxes of a pre-determined size; and
selecting a rectangle box from the four rectangle boxes to be the candidate bounding box for the respective polygon point, wherein the selected rectangle box covers the most number of data points from the cluster of 2D data points on the 2D plane compared to the rest of the four rectangle boxes;

selecting a final bounding box to be the BEV bounding box from the plurality of candidate bounding boxes, wherein the final bounding box is one of the candidate bounding boxes that covers the most number of data points from the cluster of 2D data points on the 2D plane; and
rotating the BEV bounding box based on the value of h obj around the center p center of the cluster of 2D data points on the 2D plane.

2. The method of claim 1 , wherein:
each of the four rectangle boxes has: a respective first side parallel to an x-axis of the vehicle in the vehicle coordinate system, and a respective second side parallel to a y-axis of the vehicle in the vehicle coordinate system; and
the first of the four rectangle boxes has a lower right corner coinciding with the respective polygon point, the second of the four rectangle boxes has a lower left corner coinciding with the respective polygon point, the third of the four rectangle boxes has an upper right corner coinciding with the respective polygon point, and the fourth of the four rectangle boxes has a upper left corner coinciding with the respective polygon point.

3. A processor-implemented method for generating a bounding box for an object in proximity to a vehicle, the method comprising:
receiving a three-dimensional (3D) point cloud representative of an environment;
receiving a two-dimensional (2D) image of the environment;
processing the 3D point cloud to identify an object cluster of 3D data points for a 3D object in the 3D point cloud;
processing the 2D image to detect a 2D object in the 2D image and generate information regarding the 2D object from the 2D image, wherein the information from the 2D object comprises an image heading h image of the object, an image heading uncertainty σ image 2 associated with the image heading h image of the object, a class label associated with the object, a classification score associated with the class label, and a size of the object;
determining that the 3D object and the 2D object correspond to the same object in the environment based on: the class label associated with the object, the classification score associated with the class label, and the size of the object; and
when the 3D object and the 2D object correspond to the same object in the environment:
generating a bird's eye view (BEV) bounding box for the object based on the object cluster of 3D data points and the information from the 2D image, the generating including:
mapping the object cluster of 3D data points to a cluster of 2D data points on a 2D plane in a bird's eye view (BEV) and in a vehicle coordinate system of the vehicle;
determining and storing a group of BEV polygon points on the 2D plane in the BEV, wherein the group of BEV polygon points forms a convex hull enclosing the cluster of 2D data points on the 2D plane;
determining a center p center of the cluster of 2D data points on the 2D plane;
receiving or determine, a tracked heading h track of the object and a tracked heading uncertainty σ track 2 associated with the tracked heading h track of the object;
computing and storing the estimated heading h obj of the object based on the image heading h image of the object and the tracked heading h track of the object;
computing and storing an estimated heading uncertainty σ obj 2 of the object based on the image heading uncertainty σ image 2 and the tracked heading uncertainty σ track 2 ;
rotating the cluster of 2D data points on the 2D plane around the center p center based on the estimated heading h obj ;
determining a plurality of selected polygon points from the group of BEV polygon points;
determining a plurality of candidate bounding boxes, wherein each candidate bounding box is determined based on a respective selected polygon point from the plurality of selected polygon points;
selecting a final bounding box to be the BEV bounding box from the plurality of candidate bounding boxes, wherein the final bounding box is one of the candidate bounding boxes that covers the most number of data points from the cluster of 2D data points on the 2D plane; and
rotating the final BEV bounding box based on the value of h obj around the center p center of the cluster of 2D data points on the 2D plane.

4. The method of claim 3 , wherein h obj =f (h image , h track ) and f( )is a function for computing an average value based on h image and h track .

5. The method of claim 4 , wherein

h

o
⁢
b
⁢
j

=

(

h
image

+

h
tracking

)

2

.

6. The method of claim 3 , wherein σ obj 2 =(σ image 2 , σ track 2 )and g( ) is a function for calculating an average value based on σ image 2 , and σ track 2 .

7. The

Assignees

• Taghavi Ehsan
• Nabatchian Amirhosein
• Liu Bingbing
• Huawei Tech Co Ltd

Inventors

• Taghavi Ehsan
• Nabatchian Amirhosein
• Liu Bingbing

Classifications

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V20/647 Primary

  by matching two-dimensional images to three-dimensional objects · CPC title