Sensor region of interest selection based on multisensor data

Patent metadata

Field	Value
Publication number	US-11428550-B2
Application number	US-202016808314-A
Country	US
Kind code	B2
Filing date	Mar 3, 2020
Priority date	Mar 3, 2020
Publication date	Aug 30, 2022
Grant date	Aug 30, 2022

Abstract

A system may include a first sensor of a first type and a second sensor of a second different type and having a detector. A field of view of the second sensor may be formed by a plurality of regions of interest (ROIs) defined by the detector. Control circuitry of the system may be configured to perform operations including obtaining, from the first sensor, first sensor data representing an environment, and determining, based on the first sensor data, information associated with a feature of interest within the environment. The operations may also include determining, based on the information, a particular ROI that corresponds to an expected position of the feature at a later time, obtaining a plurality of ROI sensor data from the particular ROI instead of obtaining full-resolution sensor data, and analyzing the plurality of ROI sensor data to determine one or more attributes of the feature.

First claim

What is claimed is:

1. A system comprising:
a first sensor of a first type;
a second sensor of a second type different from the first type and comprising at least one detector, wherein a field of view of the second sensor is formed by a plurality of regions of interest (ROIs) defined by the at least one detector; and
control circuitry configured to perform operations comprising:
obtaining, from the first sensor and at a first time, first sensor data representing an environment;
determining, based on the first sensor data, information associated with a feature of interest within the environment;
determining, based on the information, a particular ROI of the plurality of ROIs of the second sensor that corresponds to an expected position of the feature of interest within the environment at a second time later than the first time;
obtaining, from the second sensor, a plurality of ROI sensor data from the particular ROI instead of obtaining full-resolution sensor data that contains each respective ROI of the plurality of ROIs; and
analyzing the plurality of ROI sensor data to determine one or more attributes of the feature of interest.

2. The system of claim 1 , wherein a first subset of the control circuitry forms part of the first sensor, wherein a second subset of the control circuitry forms part of the second sensor, wherein the first subset of the control circuitry is configured to: (i) obtain the first sensor data, (ii) determine the information associated with the feature of interest, (iii) select the second sensor from a plurality of sensors of the second type based on a pose of the second sensor with respect to the environment and the expected position of the feature of interest within the environment at the second time, (iv) determine the particular ROI further based on selection of the second sensor, and (v) transmit an indication of the particular ROI to the second subset of the control circuitry by way of a peer-to-peer connection between the first sensor and the second sensor, and wherein the second subset of the control circuitry is configured to receive the indication of the particular ROI from the first subset of the control circuitry and obtain the plurality of ROI sensor data in response to reception of the indication.

3. The system of claim 1 , wherein a first subset of the control circuitry forms part of the first sensor, wherein a second subset of the control circuitry forms part of the second sensor, wherein the first subset of the control circuitry is configured to: (i) obtain the first sensor data, (ii) determine the information associated with the feature of interest and (iii) broadcast the expected position of the feature of interest to a plurality of sensors that includes the second sensor by way of a plurality of peer-to-peer connections between the first sensor and the plurality of sensors, wherein the second subset of the control circuitry is configured to determine, in response to reception of the broadcast and based on a pose of the second sensor with respect to the environment and the expected position of the feature of interest within the environment, the particular ROI of the second sensor expected to view the feature of interest at the second time and obtain the plurality of ROI sensor data in response to determining the particular ROI.

4. The system of claim 1 , wherein a first subset of the control circuitry forms part of the first sensor, wherein a second subset of the control circuitry forms part of the second sensor, and wherein the operations further comprise:
generating, by the first subset of the control circuitry and based on the information associated with the feature of interest, instructions configured to adjust a sensor parameter of the particular ROI of the second sensor, wherein the particular ROI is expected to contain the feature of interest at the second time based on the expected position;
transmitting, by the first subset of the control circuitry, the instructions to the second subset of the control circuitry by way of a peer-to-peer connection between the first sensor and the second sensor; and
adjusting, by the second subset of the control circuitry and prior to obtaining the plurality of ROI sensor data, the sensor parameter of the particular ROI according to the instructions.

5. The system of claim 4 , wherein the instructions are further based on a value of at least one sensor parameter used by the first sensor at the first time.

6. The system of claim 4 , wherein the second sensor is an image sensor, and wherein the sensor parameter comprises at least one of: (i) gain, (ii) exposure time, (iii) white balance, (iv) magnification, or (v) frame rate.

7. The system of claim 4 , further comprising:
a vehicle; and
a control system configured to control the vehicle based on data generated by at least one of the first sensor or the second sensor, wherein the transmitting of the instructions and the adjusting of the sensor parameter are performed independently of the control system.

8. The system of claim 7 , wherein the control system is configured to:
obtain, from the first sensor and prior to the first time, second sensor data representing a portion of the environment;
determine, based on the second sensor data, a value of the sensor parameter to be used by the first sensor at the first time; and
transmit the value of the sensor parameter to the first subset of the control circuitry, wherein the first subset of the control circuitry is configured to:
in response to reception of the value of the sensor parameter from the control system, adjust the sensor parameter to the value;
cause the first sensor to capture the first sensor data after adjusting the sensor parameter to the value; and
detect, based on the first sensor data, the feature of interest within the environment.

9. The system of claim 1 , wherein the one or more attributes of the feature of interest comprise one or more of: (i) geometric properties of the feature of interest, (ii) an actual position of the feature of interest within the environment at the second time, (iii) a speed of the feature of interest, (iv) an optical flow associated with the feature of interest, (v) a classification of the feature of interest, or (vi) one or more confidence values associated with results of the analyzing of the plurality of ROI sensor data.

10. The system of claim 1 , wherein the first sensor comprises a light detection and ranging (LIDAR) device, wherein the second sensor comprises an image sensor, wherein the at least one detector comprises a plurality of pixels, wherein the plurality of ROI sensor data comprise a plurality of ROI images, wherein the information associated with the feature of interest comprises a reflectivity of the feature of interest, and wherein the operations further comprise:
determining, based on the reflectivity of the feature of interest, at least one of an exposure value of an exposure parameter of the particular ROI or a gain value of a gain parameter of the particular ROI; and
prior to obtaining the plurality of ROI images, adjusting at least one of the exposure parameter of the particular ROI to the exposure value or the gain parameter of the particular ROI to the gain value.

11. The system of claim 1 , wherein the first sensor comprises at least one of (i) a light detection and ranging (LIDAR) device or (ii) a radio detection and ranging (RADAR) device, wherein the second sensor comprises an image sensor, wherein the at least one detector comprises a plurality of pixels, wherein the plurality of ROI sensor data comprise a plurality of ROI images, wherein the information associated with the feature of interest comprises a speed of the feature of in

Assignees

• Waymo Llc

Inventors

• Dharia Nirav
• Guidi Giulia
• Shepard Ralph
• Cardei Vlad
• Ion Lucian

Classifications

• H04N23/23

  from thermal infrared radiation · CPC title

• G01S17/931

  of land vehicles · CPC title

• B60W40/02

  related to ambient conditions · CPC title

• G01S13/865

  Combination of radar systems with lidar systems · CPC title

• G01S15/87

  Combinations of sonar systems · CPC title