Smart sensor with region of interest capabilities

Patent metadata

Field	Value
Publication number	US-12526547-B2
Application number	US-202218066004-A
Country	US
Kind code	B2
Filing date	Dec 14, 2022
Priority date	Sep 5, 2019
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

A method includes obtaining, from an image sensor, one or more images that represent an object, and determining a speed of the object based on the one or more images. The method also includes determining that the speed of the object exceeds a threshold speed and, based on determining that the speed of the object exceeds the threshold speed, determining a region of interest (ROI) of the image sensor expected to represent the object. The method further includes causing the image sensor to generate one or more ROI images using the ROI.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
obtaining, from an image sensor, a plurality of images that represent an object, wherein the plurality of images comprises a first image and a second image; determining a speed of the object based on the plurality of images by determining an optical flow between the first image and the second image; determining that the speed of the object exceeds a threshold speed by determining that one or more optical flow values of the optical flow exceed a threshold optical flow value; based on determining that the speed of the object exceeds the threshold speed, determining a region of interest (ROI) of the image sensor expected to represent the object; and causing the image sensor to generate one or more ROI images using the ROI.

2 . The computer-implemented method of claim 1 , wherein the image sensor is configured to allow for selection of a plurality of connected pixels to define the ROI to span a variable area of the image sensor, and wherein determining the ROI comprises:
determining the plurality of connected pixels to define the ROI.

3 . The computer-implemented method of claim 1 , wherein the image sensor comprises a plurality of ROIs, and wherein each respective ROI of the plurality of ROIs is fixed with respect to pixels of the image sensor, and wherein determining the ROI comprises:
selecting the ROI from the plurality of ROIs.

4 . The computer-implemented method of claim 1 , wherein the ROI is determined based on one or more of a size of the object, the speed of the object, a direction of motion of the object, a time difference between generating the one or more images of the plurality of images and generating the one or more ROI images, or a frame rate at which the image sensor is configured to generate the one or more ROI images.

5 . The computer-implemented method of claim 1 , wherein determining the ROI comprises:
determining a first one or more pixels of the image sensor that correspond to the one or more optical flow values; determining a second one or more pixels of the image sensor expected to represent the object based on the one or more optical flow values and respective coordinates of the first one or more pixels; and determining the ROI based on the second one or more pixels.

6 . The computer-implemented method of claim 1 , wherein the optical flow is a full-resolution optical flow, and wherein determining the full-resolution optical flow comprises:
determining a first down-sampled image based on the first image and a second down-sampled image based on the second image; determining a down-sampled optical flow between the first down-sampled image and the second down-sampled image; determining initial values of the full-resolution optical flow by upsampling the down-sampled optical flow; and updating the initial values of the full-resolution optical flow based on matching pixel values between the first image and the second image.

7 . The computer-implemented method of claim 6 , wherein each of the first down-sampled image, the second down-sampled image, and the down-sampled optical flow corresponds to a first resolution that is smaller than a resolution of each of the first image and the second image, and wherein determining the down-sampled optical flow comprises:
determining a third down-sampled image based on the first image and a fourth down-sampled image based on the second image, wherein each of the third down-sampled image and the fourth down-sampled image corresponds to a second resolution that is smaller than the first resolution; determining a second down-sampled optical flow between the third down-sampled image and the fourth down-sampled image; determining initial values of the down-sampled optical flow by upsampling the second down-sampled optical flow; and updating the initial values of the down-sampled optical flow based on matching pixel values between the first down-sampled image and the second down-sampled image.

8 . The computer-implemented method of claim 1 , wherein determining that the one or more optical flow values exceed the threshold optical flow value comprises:
determining that respective optical flow values associated with a plurality of connected pixels of the image sensor each exceed the threshold optical flow value.

9 . The computer-implemented method of claim 1 , wherein the image sensor is configured to generate the one or more ROI images at a higher frame rate than the plurality of images.

10 . The computer-implemented method of claim 1 , wherein the plurality of images comprises a full-size image generated using all pixels of the image sensor.

11 . The computer-implemented method of claim 1 , further comprising:
processing the one or more ROI images to determine one or more attributes of the object of interest.

12 . The computer-implemented method of claim 11 , wherein processing the one or more ROI images comprises:
processing the one or more ROI images using a machine learning model.

13 . The computer-implemented method of claim 1 , wherein the image sensor is provided on a first layer of an integrated circuit, and wherein the integrated circuit comprises one or more additional layers comprising circuitry configured to: (i) obtain the plurality of images from the first layer, (ii) determine the speed of the object, (iii) determine that the speed of the object exceeds the threshold speed, (iv) determine the ROI, and (v) transmit, to the first layer, instructions configured to cause the image sensor to generate the one or more ROI images.

14 . The computer-implemented method of claim 1 , further comprising:
after obtaining the one or more ROI images from the image sensor, obtaining, from the image sensor, one or more additional images that represent a second object; determining a second speed of the second object based on the one or more additional images; determining that the second speed of the second object exceeds the threshold speed; based on determining that the second speed of the second object exceeds the threshold speed, determining a second ROI of the image sensor expected to represent the second object; and causing the image sensor to generate one or more additional ROI images using the second ROI.

15 . A system comprising:
an image sensor; circuitry configured to perform operations comprising:
obtaining, from the image sensor, a plurality of images that represent an object, wherein the plurality of images comprises a first image and a second image;
determining a speed of the object based on the plurality of images by determining an optical flow between the first image and the second image;
determining that the speed of the object exceeds a threshold speed by determining that one or more optical flow values of the optical flow exceed a threshold optical flow value;
based on determining that the speed of the object exceeds the threshold speed, determining a region of interest (ROI) of the image sensor expected to represent the object; and
causing the image sensor to generate one or more ROI images using the ROI.

16 . The system of claim 15 , wherein the image sensor is configured to allow for selection of a plurality of connected pixels to define the ROI to span a variable area of the image sensor, and wherein determining the ROI comprises:
determining the plurality of connected pixels to define the ROI.

17 . The system of claim 15 , wherein determining the ROI comprises:
determining a first one or more pixels of the image sensor that correspond to the one or more optica

Assignees

• Waymo Llc

Inventors

• Dharia Nirav
• Feng Jia

Classifications

• H04N25/47

  Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data · CPC title

• H04N25/44

  by partially reading an SSIS array · CPC title

• H04N25/79

  Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors · CPC title

• H04N25/713 Primary

  Transfer or readout registers; Split readout registers or multiple readout registers · CPC title

• H04N25/78 Primary

  Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters · CPC title