Hybrid system with a structured-light stereo device and a time of flight device

Patent metadata

Field	Value
Publication number	US-10891745-B1
Application number	US-201816055355-A
Country	US
Kind code	B1
Filing date	Aug 6, 2018
Priority date	Aug 14, 2017
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

A system for real-time depth sensing includes a structured-light stereo device, a time of flight device, and a computing device. The computing device executes instructions that cause the computing device to determine depth measurements of a scene using information received by the structured-light stereo device and to determine time of flight measurements of the scene using information received by the time of flight device. The computing device executes further instructions that cause the computing device to generate a depth map using the depth measurements, to generate calibration points using the time of flight measurements, and to update the depth map using the calibration points.

First claim

What is claimed is:

1. A system for depth sensing, the system comprising:
a structured-light stereo device;
a time of flight device;
a modulator; and
a computing device including a processor configured to execute instructions stored in a memory to:
project a light pattern onto a scene, wherein modulation of the light pattern is modified based on at least one of mapping information, route information, surface conditions, or speed information from a vehicle moving through the scene,
determine depth measurements of the scene using information about the light pattern received by the structured-light stereo device,
determine time of flight measurements of the scene using information about the light pattern received by the time of flight device, wherein the light pattern is projected onto the scene using the modulator and both the structured-light stereo device and the time of flight device to determine the depth measurements and the time of flight measurements,
generate a depth map using the depth measurements,
generate calibration points using the time of flight measurements and the depth measurements, each calibration point including information related to distance between a corresponding time of flight measurement and a corresponding depth measurement, and
update the depth map using the calibration points.

2. The system of claim 1 , wherein the structured-light stereo device comprises an illuminator and a camera within a headlight space of a vehicle, on a top of the vehicle, or on a front of the vehicle.

3. The system of claim 2 , wherein the time of flight device comprises a Light Detection and Ranging (LIDAR) signal device within the headlight space of the vehicle, on the top of the vehicle, or on the front of the vehicle.

4. The system of claim 2 , wherein the instructions to update the depth map using the calibration points further comprise instructions to:
calibrate the camera within the headlight space of the vehicle, on the top of the vehicle, or on the front of the vehicle using the depth map and the calibration points.

5. The system of claim 1 , wherein the processor is further configured to execute instructions to:
synchronize projection of the light pattern using the modulator based on metadata and timing information associated with both the structured-light stereo device and the time of flight device.

6. The system of claim 1 , wherein the light pattern comprises any of a line, a beam, an array, a hybrid pattern, and a structured pattern.

7. The system of claim 1 , wherein the processor is further configured to execute instructions to:
determine the modulation of the light pattern based on an analysis of similar scenes that the vehicle has encountered prior to projection of the light pattern onto the scene.

8. The system of claim 1 , wherein the processor is further configured to execute instructions to:
generate a reflectivity map using the depth measurements and the time of flight measurements.

9. The system of claim 1 , wherein the processor is further configured to execute instructions to:
detect an object within the scene; and
target a portion of the light pattern and the time of flight measurements to only a portion of the scene that includes the object, wherein the depth map corresponds to the portion of the scene.

10. The system of claim 1 , wherein a density of the time of flight measurements is thinly dispersed across the scene.

11. A method, comprising:
projecting a light pattern onto a scene to generate a scene projection using a modulator, a structured-light stereo device, and a time of flight device,
wherein modulation of wavelengths in the light pattern is modified based on at least one of detected object information, mapping information, route information, surface conditions, or speed information from a vehicle moving through the scene;

determining depth measurements of the scene projection using information from the structured-light stereo device;
determining time of flight measurements of the scene projection using information from the time of flight device;
generating a depth map using the depth measurements;
generating calibration points using distance measurements between corresponding time of flight measurements and corresponding depth measurements;
updating the depth map using the calibration points; and
controlling operation of an automated system of the vehicle based on the depth map.

12. The method of claim 11 , further comprising:
synchronizing projection of the light pattern using the modulator and metadata and timing information associated with the structured-light stereo device and the time of flight device.

13. A method, comprising:
generating a first hybrid pattern image that includes a first amount of structured-light information and Light Detection and Ranging (LIDAR) or radar information using a structured-light stereo device and a time of flight device;
projecting the first hybrid pattern image onto a portion of a scene to generate a first projection using a modulator, the structured-light stereo device, and the time of flight device;
performing a first synchronized data capture in response to the first projection;
generating a depth map based on the first synchronized data capture;
generating a plurality of calibration points based on the LIDAR or radar information;
performing a camera calibration based on the depth map and the plurality of calibration points;
generating a second hybrid pattern image that includes a second amount of structured-light information and LIDAR or radar information using the structured-light stereo device and the time of flight device, the second amount different from the first amount, and the second amount based on a change in mapping information, route information, surface conditions, or speed information from a vehicle moving through the scene;
projecting the second hybrid pattern image onto the portion of the scene to generate a second projection using the modulator, the structured-light stereo device, and the time of flight device;
performing a second synchronized data capture in response to the second projection; and
generating an updated depth map based on the second synchronized data capture, the depth map, and a result of the camera calibration.

14. The method of claim 13 , wherein the structured-light information is based on a depth measurement and the LIDAR or radar information is based on a time of flight measurement.

15. The method of claim 13 , wherein the depth map is a stereo depth map.

16. The method of claim 13 , wherein the depth map and the plurality of calibration points are generated simultaneously.

17. The method of claim 13 , wherein each of the plurality of calibration points is calculated as a relative offset measurement between the structured-light information and the LIDAR or radar information.

18. The method of claim 11 , further comprising:
detecting an object within the scene; and
targeting a portion of the light pattern and the time of flight measurements to only a portion of the scene that includes the object, wherein the depth map corresponds to the portion of the scene.

19. The method of claim 11 , wherein the light pattern comprises any of a line, a beam, an array, a hybrid pattern, and a structured pattern.

20. The method of claim 11 , further comprising:
determining the modulation of the light pattern based on an analysis of similar scenes that the vehicle has encountered prior to projection of the light pattern onto the scene.

Assignees

• Potter Daniel E
• Remesch Bryce J
• He Chensong
• Geng Xiao
• Apple Inc

Inventors

• Potter Daniel E
• Remesch Bryce J
• He Chensong
• Geng Xiao

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/10

  using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32) · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G06T2207/30256

  Lane; Road marking · CPC title