Smart sensor scheduler

Patent metadata

Field	Value
Publication number	US-11700459-B2
Application number	US-202217651325-A
Country	US
Kind code	B2
Filing date	Feb 16, 2022
Priority date	Dec 30, 2019
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), image processing resources, and a scheduler configured to perform operations including determining a priority level for a particular ROI of the plurality of ROIs based on a feature detected by one or more image processing resources of the image processing resources within initial image data associated with the particular ROI. The operations also include selecting, based on the feature detected within the initial image data, a particular image processing resource of the image processing resources by which subsequent image data generated by the particular ROI is to be processed. The operations further include inserting, based on the priority level, the subsequent image data into a processing queue of the particular image processing resource to schedule the subsequent image data for processing by the particular image processing resource.

First claim

What is claimed is:

1. A system comprising:
an image sensor comprising a plurality of pixels that form a plurality of regions of interest (ROIs);
a first plurality of image processing resources located in a first layer of an integrated circuit;
a second plurality of image processing resources located in a second layer of the integrated circuit; and
a scheduler comprising one or more of circuitry or a processor, and configured to perform operations comprising:
identifying a particular ROI of the plurality of ROIs based on a feature detected within an initial image data associated with the particular ROI;
selecting, based on the feature detected within the initial image data, (i), from the first plurality of image processing resources, a first image processing resource by which subsequent image data generated by the particular ROI is to be processed to generate an intermediate result and (ii), from the second plurality of image processing resources, a second image processing resource by which the intermediate result is to be processed; and
scheduling (i) the subsequent image data for processing by the first image processing resource and (ii) the intermediate result for processing by the second image processing resource.

2. The system of claim 1 , wherein the first plurality of image processing resources comprises pixel-level processing circuitry configured to operate on outputs of the plurality of pixels, and wherein the second plurality of image processing resources comprises machine learning circuitry configured to operate on outputs of the pixel-level processing circuitry.

3. The system of claim 1 , wherein the plurality of pixels are located in a third layer of the integrated circuit that is disposed on a first side of the first layer of the integrated circuit, and wherein the second layer of the integrated circuit is disposed on a second side of the first layer of the integrated circuit.

4. The system of claim 1 , wherein the feature is detected by one or more image processing resources of the first plurality of image processing resources and the second plurality of image processing resources.

5. The system of claim 1 , wherein at least one image processing resource of the first image processing resource or the second image processing resource is selected further based on the at least one image processing resource being spatially co-located with the particular ROI.

6. The system of claim 1 , wherein:
scheduling the subsequent image data for processing by the first image processing resource comprises inserting the subsequent image data into a first processing queue of the first image processing resource; and
scheduling the intermediate result for processing by the second image processing resource comprises inserting the intermediate result into a second processing queue of the second image processing resource.

7. The system of claim 6 , wherein inserting the intermediate result into the second processing queue comprises:
determining (i) a first position of the subsequent image data in the first processing queue and (ii) an expected processing time of the subsequent image data by the first image processing resource; and
before the intermediate result is generated, inserting the intermediate result at a second position within the second processing queue, wherein the second position is separated in time from the first position by at least the expected processing time.

8. The system of claim 6 , wherein inserting the intermediate result into the second processing queue comprises:
determining an expiration time by which the intermediate result is to be processed by the second image processing resource; and
associating the expiration time with the intermediate result in the second processing queue, wherein the second image processing resource is configured to omit processing the intermediate result after the expiration time.

9. The system of claim 6 , wherein one or more of the first image processing resource or the second image processing resource is selected further based on an amount of data in one or more of the first processing queue or the second processing queue.

10. The system of claim 1 , wherein identifying the particular ROI comprises:
identifying the particular ROI based on a subset of the plurality of pixels, wherein the subset represents the feature detected within the initial image data.

11. The system of claim 1 , wherein the operations further comprise:
determining a priority level for the particular ROI of the plurality of ROIs based on the feature detected within the initial image data associated with the particular ROI, wherein scheduling of the subsequent image data and the intermediate result is based on the priority level.

12. The system of claim 11 , wherein determining the priority level for the particular ROI comprises:
selecting the priority level from a plurality of priority levels based on the feature detected within the initial image data, wherein each respective priority level of the plurality of priority levels is associated with at least one corresponding image feature.

13. The system of claim 11 , further comprising:
a vehicle configured to operate based on outputs of one or more of the first plurality of image processing resources or the second plurality of image processing resources, wherein the priority level for the particular ROI is determined further based on (i) a task being carried out by the vehicle and (ii) a relationship between the task and the feature detected within the initial image data.

14. The system of claim 1 , wherein at least one of the first image processing resource or the second image processing resource is selected further based on environmental conditions expected to be present at a time when the subsequent image data is generated.

15. The system of claim 1 , wherein the initial image data is generated by the particular ROI, and wherein identifying the particular ROI comprises:
determining that the feature detected within the initial image data is expected to remain within the particular ROI at a time when the subsequent image data is planned to be captured.

16. The system of claim 1 , wherein the initial image data is generated by a second ROI different from the particular ROI, and wherein identifying the particular ROI comprises:
determining that the feature detected within the initial image data is expected to move from the second ROI to the particular ROI and be viewable by the particular ROI at a time when the subsequent image data is planned to be captured.

17. The system of claim 1 , wherein one or more of the first image processing resource or the second image processing resource is selected further based on an at least one of:
(i) a first objective function configured to quantify a latency between obtaining image data by the image sensor and processing the image data, (ii) a second objective function configured to quantify a utilization of at least one of the first plurality of image processing resources or the second plurality of image processing resources, or (iii) a third objective function configured to quantify a throughput of the image data through at least one of the first plurality of image processing resources or the second plurality of image processing resources.

18. The system of claim 1 , wherein the scheduler is also configured to schedule operation of one or more of: (i) a control system of an autonomous vehicle or (ii) communicative connections between different groups of image processing resources.

19. A method comprising:
identifyi

Assignees

• Waymo Llc

Inventors

• Cardei Vlad
• Ion Lucian
• Dharia Nirav

Classifications

• H04N25/79

  Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors · CPC title

• G06V10/22

  by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/61

  based on recognised objects · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title