Sensor region of interest selection based on multisensor data

Patent metadata

Field	Value
Publication number	US-11933647-B2
Application number	US-202217872244-A
Country	US
Kind code	B2
Filing date	Jul 25, 2022
Priority date	Mar 3, 2020
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A system may include a first sensor of a first type and a second sensor of a second different type and having a detector. A field of view of the second sensor may be formed by a plurality of regions of interest (ROIs) defined by the detector. Control circuitry of the system may be configured to perform operations including obtaining, from the first sensor, first sensor data representing an environment, and determining, based on the first sensor data, information associated with a feature of interest within the environment. The operations may also include determining, based on the information, a particular ROI that corresponds to an expected position of the feature at a later time, obtaining a plurality of ROI sensor data from the particular ROI instead of obtaining full-resolution sensor data, and analyzing the plurality of ROI sensor data to determine one or more attributes of the feature.

First claim

What is claimed is:

1. A method comprising:
obtaining, by control circuitry, from a first sensor, and at a first time, first sensor data representing an environment;
determining, by the control circuitry and based on the first sensor data, information associated with a feature of interest within the environment;
determining, by the control circuitry and based on the information, a particular region of interest (ROI) of a plurality of ROIs of a second sensor, wherein a field of view of the second sensor is formed by the plurality of ROIs, and wherein the particular ROI corresponds to an expected position of the feature of interest within the environment at a second time later than the first time; and
obtaining, by the control circuitry and from the second sensor, a plurality of ROI sensor data from the particular ROI.

2. The method of claim 1 , wherein:
a first subset of the control circuitry forms part of the first sensor;
a second subset of the control circuitry forms part of the second sensor;
the first subset of the control circuitry is configured to:
obtain the first sensor data;
determine the information associated with the feature of interest;
determine the particular ROI based on a pose of the second sensor with respect to the environment and the expected position of the feature of interest within the environment at the second time; and
transmit an indication of the particular ROI to the second subset of the control circuitry by way of a peer-to-peer connection between the first sensor and the second sensor; and

the second subset of the control circuitry is configured to:
receive the indication of the particular ROI from the first subset of the control circuitry; and
obtain the plurality of ROI sensor data based on reception of the indication.

3. The method of claim 2 , wherein determining the particular ROI further comprises:
selecting the second sensor from a plurality of sensors based on a pose of the second sensor with respect to the environment and the expected position of the feature of interest within the environment at the second time.

4. The method of claim 1 , wherein:
a first subset of the control circuitry forms part of the first sensor;
a second subset of the control circuitry forms part of the second sensor;
the first subset of the control circuitry is configured to:
obtain the first sensor data;
determine the information associated with the feature of interest; and
transmit the information to the second subset of the control circuitry by way of a peer-to-peer connection between the first sensor and the second sensor; and

the second subset of the control circuitry is configured to:
determine, based on reception of the information, the particular ROI of the second sensor; and
obtain the plurality of ROI sensor data based on determining the particular ROI.

5. The method of claim 4 , wherein:
transmitting the information to the second subset of the control circuitry by way of the peer-to-peer connection between the first sensor and the second sensor comprises transmitting the expected position of the feature of interest to a plurality of sensors that includes the second sensor by way of a plurality of peer-to-peer connections between the first sensor and the plurality of sensors; and
determining the particular ROI further comprises determining the particular ROI in response to reception of the information and based on (i) a pose of the second sensor with respect to the environment and (ii) the expected position of the feature of interest within the environment.

6. The method of claim 1 , further comprising:
generating, based on (i) the information associated with the feature of interest and (ii) a value of at least one sensor parameter used by the first sensor at the first time, instructions configured to adjust a sensor parameter of the particular ROI of the second sensor;
transmitting the instructions to the second sensor; and
prior to obtaining the plurality of ROI sensor data, adjusting the sensor parameter of the particular ROI according to the instructions.

7. The method of claim 6 , wherein the transmitting of the instructions and the adjusting of the sensor parameter are performed by the control circuitry independently of a control system of a vehicle, wherein the first sensor and the second sensor are each provided on the vehicle, and wherein, the control system is configured to control the vehicle based on data generated by at least one of the first sensor or the second sensor.

8. The method of claim 7 , further comprising:
obtaining, by the control system from the first sensor and prior to the first time, second sensor data representing a portion of the environment;
determining, by the control system and based on the second sensor data, the value of the at least one sensor parameter to be used by the first sensor at the first time;
transmitting, by the control system, the value of the at least one sensor parameter to the first sensor;
in response to reception of the value of the at least one sensor parameter from the control system, adjusting, using the control circuitry, the at least one sensor parameter to the value;
causing, by the control circuitry, the first sensor to capture the first sensor data after adjusting the at least one sensor parameter to the value; and
detecting, based on the first sensor data and using the control circuitry, the feature of interest within the environment.

9. The method of claim 1 , further comprising:
analyzing, by the control circuitry, the plurality of ROI sensor data to determine one or more attributes of the feature of interest.

10. The method of claim 9 , wherein the one or more attributes of the feature of interest comprise one or more of: (i) geometric properties of the feature of interest, (ii) an actual position of the feature of interest within the environment at the second time, (iii) a speed of the feature of interest, (iv) an optical flow associated with the feature of interest, (v) a classification of the feature of interest, or (vi) one or more confidence values associated with results of the analyzing of the plurality of ROI sensor data.

11. The method of claim 1 , further comprising:
obtaining, from the second sensor and at the first time, a full-resolution sensor data that contains each respective ROI of the plurality of ROIs, wherein the information associated with the feature of interest is determined further based on the full-resolution sensor data, wherein the particular ROI is determined further based on the full-resolution sensor data, and wherein the plurality of ROI sensor data is obtained from the particular ROI instead of obtaining additional full-resolution sensor data.

12. The method of claim 1 , wherein the control circuitry comprises neural network circuitry, and wherein determining the information associated with the feature of interest comprises:
analyzing the first sensor data using the neural network circuitry; and
generating, by way of the neural network circuitry, neural network output data related to results of the analyzing of the first sensor data.

13. The method of claim 1 , wherein the first sensor is of a same type as the second sensor.

14. The method of claim 1 , wherein the first sensor is of a first type, and wherein the second sensor is of a second type different from the first type.

15. The method of claim 1 , wherein the second sensor comprises an image sensor, wherein the plurality of ROI sensor data comprise a plurality of ROI images, wherein the information associated with the feature of interest comprises a reflectivity of the feature of in

Assignees

• Waymo Llc

Inventors

• Dharia Nirav
• Guidi Giulia
• Shepard Ralph
• Cardei Vlad
• Ion Lucian

Classifications

• H04N23/23

  from thermal infrared radiation · CPC title

• G01D5/35383 Primary

  using multiple sensor devices using multiplexing techniques · CPC title

• G01S13/89

  for mapping or imaging · CPC title

• H04N5/33

  Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) · CPC title

• G01S7/4004

  of parts of a radar system · CPC title