Three-dimensional bounding box from two-dimensional image and point cloud data

Patent metadata

Field	Value
Publication number	US-11216971-B2
Application number	US-201916557997-A
Country	US
Kind code	B2
Filing date	Aug 30, 2019
Priority date	Sep 22, 2017
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

A three-dimensional bounding box is determined from a two-dimensional image and a point cloud. A feature vector associated with the image and a feature vector associated with the point cloud may be passed through a neural network to determine parameters of the three-dimensional bounding box. Feature vectors associated with each of the points in the point cloud may also be determined and considered to produce estimates of the three-dimensional bounding box on a per-point basis.

First claim

What we claim is:

1. A computer-implemented method comprising:
receiving sensor data comprising a plurality of measurements of an environment;
inputting at least a portion of the sensor data into a machine learned model;
determining, as a first feature vector and based at least in part on a first portion of the machine learned model, a first set of values associated with a measurement of the plurality of measurements;
determining, as a second feature vector and based at least in part on a second portion of the machine learned model, a second set of values associated with the plurality of measurements;
receiving image data from an image sensor;
determining, as a third feature vector and based at least in part on a third portion of the machine learned model, a third set of values associated with a portion of the image data;
combining, as a combined feature vector, the first feature vector, the second feature vector, and the third feature vector;
inputting the combined feature vector into a fourth portion of the machine learned model; and
receiving, from the fourth portion of the machine learned model, information associated with an object represented in the sensor data.

2. The computer-implemented method of claim 1 , wherein determining, as the third feature vector, the third set of values associated with the portion of the image data comprises:
determining a portion of the image data associated with the object;
determining, based at least in part on the portion of the image data associated with the object, a subset of the sensor data associated with the portion of the image data;
inputting the portion of the image data into a fifth fourth portion of the machine learned model; and
receiving, from the fifth portion of the machine learned model, an appearance feature vector;
wherein combining the first feature vector, the second feature vector, and the third feature vector comprises combining the appearance feature vector with the first feature vector and the second feature vector,
wherein inputting the sensor data into the machine learned model comprises inputting the subset of sensor data into the machine learned model, and
wherein the information associated with the object is further based on the appearance feature vector.

3. The computer-implemented method of claim 1 , wherein:
combining the first feature vector and the second feature vector comprises concatenating the first feature vector and the second feature vector.

4. The computer-implemented method of claim 1 , wherein the plurality of measurements comprises a plurality of Light Detection and Ranging (“LiDAR”) measurements.

5. The computer-implemented method of claim 1 , wherein the information associated with the object comprises a plurality of points that define a three-dimensional bounding box associated with the object.

6. The computer-implemented method of claim 5 , wherein the sensor data comprises point cloud data, the computer-implemented method further comprising:
determining, for a first point in the point cloud data, a first set of offsets corresponding to first estimated positions of corners of a first candidate three-dimensional bounding box relative to the first point; and
determining a confidence value associated with the first candidate three-dimensional bounding box.

7. The computer-implemented method of claim 6 , further comprising:
determining, for a second point of the plurality of points, a second set off offsets and a second confidence score, the second set of offsets corresponding to second estimated positions of the corners of a second candidate three-dimensional bounding box relative to the second point,
wherein the plurality of points that define the three-dimensional bounding box correspond to the first estimated positions based on the first confidence score being higher than the second confidence score.

8. The computer-implemented method of claim 1 , wherein the third portion of the machine learned model is trained using a regression loss.

9. The computer-implemented method of claim 1 , further comprising:
controlling an autonomous vehicle to navigate relative to the object.

10. The computer-implemented method of claim 1 , wherein:
the first feature vector comprises a local feature vector extracted from a first processing algorithm; and
the second feature vector comprises a global feature vector extracted from a second processing algorithm.

11. A system comprising:
one or more processors; and
non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to:
input sensor data into a machine learned model, the sensor data including a plurality of measurements;
determine, based on a first portion of the machine learned model, a first feature vector, the first feature comprising a first set of values associated with a first measurement of the plurality of measurements;
determine, based on a second portion of the machine learned model, a second feature vector, the second feature vector comprising a second set of values associated with the plurality of measurements;
receive image data from an image sensor;
determine, based on a third portion of the machine learned model, a third feature vector, the third feature vector comprising a third set of values associated with a portion of the image data;
combine the first feature vector, the second feature vector, and the third feature vector as a combined feature vector;
input the combined feature vector into a fourth portion of the machine learned model; and
receive, from the fourth portion of the machine learned model, information associated with an object represented in the sensor data.

12. The system of claim 11 , wherein the instructions to determine the third feature vector comprise further instructions to further cause the system to:
determine a portion of the image data associated with the object;
determine a subset of the sensor data associated with the portion of the image data;
input the portion of the image data into a fifth portion of the machine learned model; and
receive, from the fourth portion of the machine learned model, an appearance feature vector comprising a third set of values;
wherein combining the first feature vector, the second feature vector, and the third feature vector comprises combining the appearance feature vector with the first feature vector and the second feature vector, and
wherein the information associated with the object is further based on the appearance feature vector.

13. The system of claim 11 , wherein the information associated with the object comprises a plurality of points that define a three-dimensional bounding box associated with the object.

14. The system of claim 13 , wherein the sensor data comprises point cloud data, the instructions further causing the system to:
determine, for a first point in the point cloud data, a first set of offsets corresponding to estimated positions of corners of a candidate three-dimensional bounding box relative to the first point; and
determine a confidence value associated with the first candidate three-dimensional bounding box.

15. The system of claim 14 , wherein the plurality of points that define the three-dimensional bounding box is based at least in part on the estimated positions and the confidence value.

16. A system comprising:
an autonomous vehicle configured to operate in an environment;
a sensor configured to generate point cloud data corresponding to the environment;
an image sensor configured to capture image data of the environment;
one or mo

Assignees

• Zoox Inc

Inventors

• Xu Danfei
• Anguelov Dragomir Dimitrov
• Jain Ashesh

Classifications

• G01S7/417 Primary

  involving the use of neural networks · CPC title

• G06V10/806

  of extracted features · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06F18/253

  of extracted features · CPC title