Smart sensor implementations of region of interest operating modes

Patent metadata

Field	Value
Publication number	US-11756283-B2
Application number	US-202017123440-A
Country	US
Kind code	B2
Filing date	Dec 16, 2020
Priority date	Dec 16, 2020
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), and configured to operate at a frame rate higher than a threshold rate. The system also includes an image processing resource. The system further includes control circuitry configured to perform operations that include obtaining, from the image sensor, a full-resolution image of an environment. The full-resolution image contains each respective ROI of the plurality of ROIs. The operations also include selecting a particular ROI based on the full-resolution image, and detecting an object of interest in the particular ROI. The operations include determining a mode of operation by which subsequent image data generated by the particular ROI is to be processed. The operations further include processing, based on the mode of operation and the frame rate, the image data comprising a plurality of ROI images of the object of interest.

First claim

What is claimed is:

1. A system comprising:
an image sensor comprising a plurality of pixels that form a plurality of regions of interest (ROIs), wherein the image sensor is configured to operate at a frame rate higher than a threshold rate;
an image processing resource; and
control circuitry configured to perform operations comprising:
obtaining, from the image sensor, a full-resolution image of an environment, wherein the full-resolution image contains each respective ROI of the plurality of ROIs;
selecting a particular ROI based on the full-resolution image;
detecting at least one object of interest in the particular ROI;
determining a mode of operation by which subsequent image data generated by the particular ROI is to be processed, and
processing, based on the mode of operation and the frame rate, the image data comprising a plurality of ROI images of the at least one object of interest.

2. The system of claim 1 , wherein the processing, based on the mode of operation and the frame rate, of the image data, comprises generating, during one or more next duty cycles, the plurality of ROI images of the at least one object of interest instead of obtaining an additional full-resolution image.

3. The system of claim 1 , wherein the processing, based on the mode of operation and the frame rate, of the image data, comprises generating, during one or more next duty cycles, the plurality of ROIs and an additional full-resolution image comprising one or more objects of interest.

4. The system of claim 1 , wherein the processing, based on the mode of operation and the frame rate, of the image data, comprises generating, during one or more next duty cycles, a plurality of additional full-resolution images comprising one or more objects of interest.

5. The system of claim 1 , further comprising a vehicle, and wherein the mode of operation is based on a position of the image sensor on the vehicle.

6. The system of claim 1 , wherein a number of the plurality of ROI images of the at least one object of interest is based on the frame rate.

7. The system of claim 1 , wherein the image sensor is configured with a predetermined mode of operation.

8. The system of claim 1 , wherein the processing of the image data comprises performing one or more image processing tasks on the plurality of ROI images of the at least one object of interest.

9. The system of claim 1 , wherein the operations further comprise:
comparing a distance between the image sensor and the at least one object of interest represented within the full-resolution image to a threshold distance; and
based on results of comparing the distance to the threshold distance, obtaining the plurality of ROI images of the at least one object of interest instead of obtaining an additional full-resolution image.

10. The system of claim 1 , wherein the operations further comprise:
comparing a speed of the at least one object of interest represented within the full-resolution image to a threshold speed; and
based on results of comparing the speed to the threshold speed, obtaining the plurality of ROI images of the at least one object of interest instead of obtaining an additional full-resolution image.

11. The system of claim 1 , wherein the operations further comprise:
providing, to a server, the plurality of ROI images of the at least one object of interest and a full-resolution image comprising the at least one object of interest.

12. The system of claim 1 , wherein the full-resolution image is obtained at a first time, and wherein processing the image data comprises:
transmitting, to a second image sensor, the image data of the at least one object of interest within the environment;
determining the particular ROI from a plurality of ROIs of the second image sensor, wherein the particular ROI corresponds to an expected position of the at least one object of interest within the environment at a second time later than the first time; and
obtaining, from the second sensor, a plurality of ROI sensor data from the particular ROI instead of obtaining full-resolution sensor data that contains each respective ROI of the plurality of ROIs.

13. The system of claim 12 , wherein a first subset of the control circuitry forms part of the image sensor, wherein a second subset of the control circuitry forms part of the second image sensor, wherein the first subset of the control circuitry is configured to: (i) obtain the full-resolution image, (ii) determine information associated with the at least one object of interest, (iii) select the second image sensor from a plurality of image sensors based on a pose of the second image sensor with respect to the environment and the expected position of the at least one object of interest within the environment at the second time, (iv) determine the particular ROI further based on selection of the second image sensor, and (v) transmit an indication of the particular ROI to the second subset of the control circuitry by way of a peer-to-peer connection between the image sensor and the second image sensor, and wherein the second subset of the control circuitry is configured to receive the indication of the particular ROI from the first subset of the control circuitry and obtain the plurality of ROI sensor data in response to reception of the indication.

14. The system of claim 12 , wherein a first subset of the control circuitry forms part of the image sensor, wherein a second subset of the control circuitry forms part of the second image sensor, wherein the first subset of the control circuitry is configured to: (i) obtain the full-resolution image, (ii) determine information associated with the at least one object of interest, and (iii) broadcast the expected position of the at least one object of interest to a plurality of image sensors that includes the second image sensor by way of a plurality of peer-to-peer connections between the image sensor and the plurality of image sensors, wherein the second subset of the control circuitry is configured to determine, in response to reception of the broadcast and based on a pose of the second image sensor with respect to the environment and the expected position of the at least one object of interest within the environment, the particular ROI of the second image sensor expected to view the at least one object of interest at the second time and obtain the plurality of ROI sensor data in response to determining the particular ROI.

15. The system of claim 12 , further comprising:
a vehicle; and
a control system configured to control the vehicle based on data generated by the image sensor, wherein the transmitting of the image data, the determining of the particular ROI, and the obtaining of the plurality of ROI sensor data are performed independently of the control system.

16. The system of claim 1 , wherein the detecting of the at least one object of interest comprises determining one or more of: (i) geometric properties of the at least one object of interest, (ii) an actual position of the at least one object of interest within the environment, (iii) a speed of the at least one object of interest, (iv) an optical flow associated with the at least one object of interest, (v) a classification of the at least one object of interest, or (vi) one or more confidence values associated with results of the processing of the image data.

17. The system of claim 1 , wherein the control circuitry comprises neural network circuitry, and wherein processing of the image data comprises:
analyzing the image data using the neural network circuitry; and
generating, by way of the

Assignees

• Waymo Llc

Inventors

• Dharia Nirav
• Ion Lucian

Classifications

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N25/443

  by reading pixels from selected two-dimensional [2D] regions of the array, e.g. for windowing or digital zooming · CPC title

• H04N23/635

  Region indicators; Field of view indicators · CPC title

• H04N23/632

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title