Multi-modal sensor data association architecture

Patent metadata

Field	Value
Publication number	US-2021343022-A1
Application number	US-202117373550-A
Country	US
Kind code	A1
Filing date	Jul 12, 2021
Priority date	Apr 16, 2019
Publication date	Nov 4, 2021
Grant date	—

Abstract

A machine-learning architecture may be trained to determine point cloud data associated with different types of sensors with an object detected in an image and/or generate a three-dimensional region of interest (ROI) associated with the object. In some examples, the point cloud data may be associated with sensors such as, for example, a lidar device, radar device, etc.

First claim

What is claimed is:

1 . A method comprising:
receiving first sensor data associated with a first type of sensor, the first sensor data representing a portion of an environment surrounding an autonomous vehicle; receiving second sensor data associated with a second type of sensor, the second sensor data representing a same portion or different portion of the environment as the portion represented by the first sensor data; receiving an object detection, wherein the object detection identifies an object in one or more images; determining, based at least in part on the object detection, a first subset of the first sensor data and a second subset of the second sensor data; inputting the first subset of the first sensor data into a first subnetwork; inputting the second subset of the second sensor data into a second subnetwork; receiving a first output from the first subnetwork and a second output from the second subnetwork; combining, as a combined output, the first output and the second output; inputting a first portion of the combined output into a third subnetwork and a second portion of the combined output into a fourth subnetwork; and receiving a first map from the third subnetwork and a second map from the fourth subnetwork, wherein:
the first map indicates at least a first probability that a first point of the first sensor data is associated with the object, and
the second map indicates at least a second probability that a second point of the second sensor data is associated with the object.

Assignees

• Zoox Inc

Inventors

• Cohen Joshua Kriser
• Pradhan Sabeek Mani
• Kovacs Balazs
• Sloan Cooper Stokes

Classifications

• G01S7/417 Primary

  involving the use of neural networks · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title