Hybrid time-of-flight and imager module

Patent metadata

Field	Value
Publication number	US-12032097-B2
Application number	US-202318171883-A
Country	US
Kind code	B2
Filing date	Feb 21, 2023
Priority date	Jul 31, 2018
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

The present disclosure relates to systems and methods that provide both an image of a scene and depth information for the scene. An example system includes at least one time-of-flight (ToF) sensor and an imaging sensor. The ToF sensor and the imaging sensor are configured to receive light from a scene. The system also includes at least one light source and a controller that carries out operations. The operations include causing the at least one light source to illuminate at least a portion of the scene with illumination light according to an illumination schedule. The operations also include causing the at least one ToF sensor to provide information indicative of a depth map of the scene based on the illumination light. The operations additionally include causing the imaging sensor to provide information indicative of an image of the scene based on the illumination light.

First claim

What is claimed is:

1. A system comprising:
at least one time-of-flight (ToF) sensor;
an imaging sensor, wherein the at least one ToF sensor and the imaging sensor are configured to receive light from a scene comprising a spatial sector, wherein the spatial sector comprises a portion of an environment around a vehicle; and
a controller that carries out operations, the operations comprising:
receiving illumination light from at least a portion of the scene, wherein the illumination light is based on an illumination schedule;
causing the at least one ToF sensor to provide information indicative of a depth map of the scene based on the illumination light;
causing the imaging sensor to provide information indicative of an image of the scene based on the illumination light;
determining, based on a plurality of images of the scene, an optical flow representation of the scene; and
adjusting at least one operating parameter of the ToF sensor or the illumination light based on the optical flow representation of the scene.

2. The system of claim 1 , wherein the illumination schedule comprises at least one of: a predetermined light pulse repetition rate, a predetermined light pulse duration, a predetermined light pulse intensity, or a predetermined light pulse duty cycle.

3. The system of claim 1 , wherein the operations further comprise causing at least one light source to illuminate at least the portion of the scene with the illumination light according to the illumination schedule.

4. The system of claim 1 , wherein the operations further comprise determining a high resolution depth map of the scene based on the depth map of the scene and the image of the scene.

5. The system of claim 1 , wherein the at least one ToF sensor and the imaging sensor are coupled to a common substrate.

6. The system of claim 1 , wherein the at least one ToF sensor is operable to provide a distance based point map of the scene.

7. The system of claim 1 , wherein the operations further comprise determining at least one inference about the scene based on the depth map of the scene and the image of the scene.

8. The system of claim 7 , wherein the at least one inference comprises information about one or more objects in the environment of the vehicle or an operating context of the vehicle.

9. The system of claim 7 , wherein the controller comprises at least one deep neural network, wherein the determining the at least one inference is performed by the at least one deep neural network.

10. The system of claim 1 , wherein the at least one operating parameter of the ToF sensor includes one of: a modulation frequency, an illumination intensity, and a refresh rate.

11. A method comprising:
receiving illumination light from at least a portion of a scene, wherein the illumination light is based on an illumination schedule;
causing a time-of-flight (ToF) sensor to provide information indicative of a depth map of the scene based on the illumination light;
causing an imaging sensor to provide information indicative of an image of the scene based on the illumination light;
determining, based on a plurality of images of the scene, an optical flow representation of the scene; and
adjusting at least one operating parameter of the ToF sensor or the illumination light based on the optical flow representation of the scene.

12. The method of claim 11 , wherein the illumination schedule comprises at least one of: a predetermined light pulse repetition rate, a predetermined light pulse duration, a predetermined light pulse intensity, or a predetermined light pulse duty cycle.

13. The method of claim 11 , further comprising causing at least one light source to illuminate at least the portion of the scene with the illumination light according to the illumination schedule.

14. The method of claim 11 , further comprising determining a high-resolution depth map of the scene based on the depth map of the scene and the image of the scene.

15. The method of claim 11 , further comprising determining at least one inference about the scene based on the depth map of the scene and the image of the scene.

16. The method of claim 15 , wherein the at least one inference comprises information about one or more objects in an environment of a vehicle or an operating context of the vehicle.

17. The method of claim 15 , wherein determining the at least one inference is performed by at least one deep neural network.

18. The method of claim 11 , further comprising adjusting the illumination schedule based on an amount of ambient light or a time of day.

19. The method of claim 11 further comprising:
comparing the image of the scene and the depth map;
based on the comparison, determining at least one range-aliased artifact in the depth map; and
providing an updated depth map based on the determined at least one range-aliased artifact.

20. The method of claim 11 further comprising:
determining, based on the image of the scene, a region of interest; and
adjusting at least one operating parameter of the ToF sensor based on an object within the region of interest.

Assignees

• Waymo Llc

Inventors

• Onal Caner
• Schleuning David
• Hermalyn Brendan
• Verghese Simon
• Mccauley Alex
• White Brandyn
• Zhilinsky Ury

Classifications

• G01S17/931

  of land vehicles · CPC title

• G01S7/4816

  of receivers alone · CPC title

• G01S7/4804

  Auxiliary means for detecting or identifying lidar signals or the like, e.g. laser illuminators · CPC title

• G06T2207/10044

  Radar image · CPC title

• G06T1/0007

  Image acquisition · CPC title