Three-dimensional bounding box from two-dimensional image and point cloud data

Patent metadata

Field	Value
Publication number	US-2020005485-A1
Application number	US-201916557997-A
Country	US
Kind code	A1
Filing date	Aug 30, 2019
Priority date	Sep 22, 2017
Publication date	Jan 2, 2020
Grant date	—

Abstract

A three-dimensional bounding box is determined from a two-dimensional image and a point cloud. A feature vector associated with the image and a feature vector associated with the point cloud may be passed through a neural network to determine parameters of the three-dimensional bounding box. Feature vectors associated with each of the points in the point cloud may also be determined and considered to produce estimates of the three-dimensional bounding box on a per-point basis.

First claim

1 . (canceled)

2 . A computer-implemented method comprising:
receiving sensor data comprising a plurality of measurements of an environment; inputting at least a portion of the sensor data into a machine learned model; determining, as a first feature vector and based at least in part on a first portion of the machine learned model, a first set of values associated with a measurement of the plurality of measurements; determining, as a second feature vector and based at least in part on a second portion of the machine learned model, a second set of values associated with the plurality of measurements; combining, as a combined feature vector, the first feature vector and the second feature vector; inputting the combined feature vector into a third portion of the machine learned model; and receiving, from the third portion of the machine learned model, information associated with an object represented in the sensor data.

3 . The computer-implemented method of claim 2 , further comprising:
receiving, from an image sensor, image data of the environment; determining a portion of the image data associated with the object; determining, based at least in part on the portion of the image data associated with the object, a subset of the sensor data associated with the portion of the image data; inputting the portion of the image data into a fourth portion of the machine learned model; receiving, from the fourth portion of the machine learned model, an appearance feature vector; and inputting the appearance feature vector into the third portion of the machine learned model with the combined feature vector, wherein inputting the sensor data into the machine learned model comprises inputting the subset of sensor data into the machine learned model, and wherein the information associated with the object is further based on the appearance feature vector.

4 . The computer-implemented method of claim 2 , wherein:
combining the first feature vector and the second feature vector comprises concatenating the first feature vector and the second feature vector.

5 . The computer-implemented method of claim 2 , wherein the plurality of measurements comprises a plurality of LiDAR measurements.

6 . The computer-implemented method of claim 2 , wherein the information associated with the object comprises a plurality of points that define a three-dimensional bounding box associated with the object.

7 . The computer-implemented method of claim 6 , wherein the sensor data comprises point cloud data, the computer-implemented method further comprising:
determining, for a first point in the point cloud data, a first set of offsets corresponding to first estimated positions of corners of a first candidate three-dimensional bounding box relative to the first point; and determining a confidence value associated with the first candidate three-dimensional bounding box.

8 . The computer-implemented method of claim 7 , further comprising:
determining, for a second point of the plurality of points, a second set off offsets and a second confidence score, the second set of offsets corresponding to second estimated positions of the corners of a second candidate three-dimensional bounding box relative to the second point, wherein the plurality of points that define the three-dimensional bounding box correspond to the first estimated positions based on the first confidence score being higher than the second confidence score.

9 . The computer-implemented method of claim 2 , wherein the second portion of the machine learned model is trained using a regression loss.

10 . The computer-implemented method of claim 2 , further comprising:
controlling an autonomous vehicle to navigate relative to the object.

11 . The computer-implemented method of claim 2 , wherein:
the first feature vector comprises a local feature vector extracted from a first layer of a neural network; and the second feature vector comprises a global feature vector extracted from a second layer of the neural network.

12 . A system comprising:
one or more processors; and non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to:
input sensor data into a machine learned model, the sensor data including a plurality of measurements;
determine, based on a first portion of the machine learned model, a first feature vector, the first feature comprising a first set of values associated with a first measurement of the plurality of measurements;
determine, based on a second portion of the machine learned model, a second feature vector, the second feature vector comprising a second set of values associated with the plurality of measurements;
combine the first feature vector and the second feature vector as a combined feature vector;
input the combined feature vector into a third portion of the machine learned model; and
receive, from the third portion of the machine learned model, information associated with an object represented in the sensor data.

13 . The system of claim 12 , wherein the instructions further cause the system to:
receive, from an image sensor, image data of the environment; determine a portion of the image data associated with the object; determine a subset of the sensor data associated with the portion of the image data; input the portion of the image data into a fourth portion of the machine learned model; receive, from the fourth portion of the machine learned model, an appearance feature vector comprising a third set of values; and input the appearance feature vector into the third portion of the machine learned model with the combined feature vector, wherein the information associated with the object is further based on the appearance feature vector.

14 . The system of claim 12 , wherein the information associated with the object comprises a plurality of points that define a three-dimensional bounding box associated with the object.

15 . The system of claim 14 , wherein the sensor data comprises point cloud data, the instructions further causing the system to:
determine, for a first point in the point cloud data, a first set of offsets corresponding to estimated positions of corners of a candidate three-dimensional bounding box relative to the first point; and determine a confidence value associated with the first candidate three-dimensional bounding box.

16 . The system of claim 12 , wherein the plurality of points that define the three-dimensional bounding box is based at least in part on the estimated positions and the confidence value.

17 . A system comprising:
an autonomous vehicle configured to operate in an environment; a sensor configured to generate point cloud data corresponding to the environment; one or more processors: and non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform acts comprising:
inputting sensor data into a machine learned model, the sensor data including a plurality of measurements;
determining, based on a first portion of the machine learned model, a first feature vector, the first feature comprising a first set of values associated with a first measurement of the plurality of measurements;
determining, based on a second portion of the machine learned model, a second feature vector, the second feature vector comprising a second set of values associated with the plurality of measurements;
combining the first feature vector and the second feature vector as a combined feature vecto

Assignees

• Zoox Inc
• Zook Inc

Inventors

• Xu Danfei
• Anguelov Dragomir Dimitrov
• Jain Ashesh

Classifications

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/30236

  Traffic on road, railway or crossing · CPC title

• G01S7/417 Primary

  involving the use of neural networks · CPC title

• G06V10/806

  of extracted features · CPC title

• G06V10/82

  using neural networks · CPC title