Multi-sensor depth mapping

Patent metadata

Field	Value
Publication number	US-12590798-B2
Application number	US-202217688933-A
Country	US
Kind code	B2
Filing date	Mar 8, 2022
Priority date	May 11, 2021
Publication date	Mar 31, 2026
Grant date	Mar 31, 2026

Abstract

Depth mapping apparatus includes an illumination assembly, which directs modulated optical radiation toward a target scene, and a camera, which captures a two-dimensional image of the target scene. A range sensor senses respective times of flight of photons reflected from a matrix of locations disposed across the target scene. A processor derives first depth coordinates of the matrix of locations responsively to the respective times of flight, derives second depth coordinates of the matrix of locations responsively to a transverse disparity between features in the two-dimensional image and corresponding reference features in a reference image, computes a disparity correction function based on a difference between the first and the second depth coordinates at the matrix of locations, corrects the transverse disparity between the two-dimensional image and the reference image using the disparity correction function, and generates a depth map of the target scene based on the corrected transverse disparity.

First claim

The invention claimed is:

1 . Depth mapping apparatus, comprising:
an illumination assembly, which is configured to direct modulated optical radiation toward a target scene; a camera, which is configured to capture a two-dimensional image of the target scene; a range sensor, which is configured to sense respective times of flight of photons reflected from a matrix of locations disposed across the target scene; and a processor, which is configured to derive first depth coordinates of the matrix of locations responsively to the respective times of flight, to derive second depth coordinates of the matrix of locations responsively to a transverse disparity between features in the two-dimensional image and corresponding reference features in a reference image, to compute a disparity correction function based on a difference between the first and the second depth coordinates at the matrix of locations, to correct the transverse disparity between the two-dimensional image and the reference image using the disparity correction function, and to generate a depth map of the target scene based on the corrected transverse disparity, wherein the processor is configured to compute the disparity correction function by calculating respective local disparity corrections at the matrix of the locations and interpolating and extrapolating from the local disparity corrections to generate the disparity correction function over an entire field of view of the camera.

2 . The apparatus according to claim 1 , wherein the illumination assembly is configured to project a pattern of spatially modulated radiation onto the target scene, and the reference image comprises a reference pattern, and wherein the processor is configured to measure the transverse disparity between the projected pattern captured in the two-dimensional image and the reference pattern.

3 . The apparatus according to claim 1 , wherein the camera comprises a stereoscopic pair of first and second image sensors, wherein the first image sensor captures the two-dimensional image, and the second image sensor captures the reference image.

4 . The apparatus according to claim 1 , wherein the illumination assembly is configured to direct temporally modulated optical radiation toward the target scene, and the range sensor is configured to sense the respective times of flight responsively to a temporal modulation of the reflected photons.

5 . The apparatus according to claim 4 , wherein the temporally modulated radiation comprises a sequence of pulses of the optical radiation, and the range sensor is configured to measure the times of flight of the pulses.

6 . The apparatus according to claim 4 , wherein the temporally modulated radiation is modulated with a carrier waveform at a selected carrier frequency, and the range sensor is configured to measure a phase of the carrier frequency in the reflected photons.

7 . The apparatus according to claim 1 , wherein the range sensor comprises an array of sensing elements and objective optics, which image each of the matrix of the locations onto a respective set of one or more of the sensing elements.

8 . The apparatus according to claim 1 , wherein the processor is configured to combine the local disparity corrections computed over different areas of the field of view in different target scenes in order to generate and update the disparity correction function.

9 . The apparatus according to claim 1 , wherein the processor is configured to store the disparity correction function computed over a first target scene and to apply the disparity correction function in generating depth maps over further target scenes.

10 . The apparatus according to claim 1 , wherein the processor is configured to compute, responsively to the difference between the first and the second depth coordinates at the matrix of locations, a global correction to one or more parameters of the camera and to apply the global correction in generating the disparity correction function.

11 . Depth mapping apparatus, comprising:
an illumination assembly, which is configured to direct modulated optical radiation toward a target scene; a camera, which is configured to capture a two-dimensional image of the target scene; a range sensor, which is configured to sense respective times of flight of photons reflected from a matrix of locations disposed across the target scene; and a processor, which is configured to derive first depth coordinates of the matrix of locations responsively to the respective times of flight, to derive second depth coordinates of the matrix of locations responsively to a transverse disparity between features in the two-dimensional image and corresponding reference features in a reference image, to compute a disparity correction function based on a difference between the first and the second depth coordinates at the matrix of locations, to correct the transverse disparity between the two-dimensional image and the reference image using the disparity correction function, and to generate a depth map of the target scene based on the corrected transverse disparity, wherein the processor is configured to compute a range correction function over a first span of distances from the apparatus by comparing the first depth coordinates to the second depth coordinates in the depth map, and to apply the range correction function in correcting the first depth coordinates over a second span of distances, greater than the first span.

12 . A method for depth mapping, comprising:
directing modulated optical radiation toward a target scene; capturing a two-dimensional image of the target scene; sensing respective times of flight of photons reflected from a matrix of locations disposed across the target scene; deriving first depth coordinates of the matrix of locations responsively to the respective times of flight; deriving second depth coordinates of the matrix of locations responsively to a transverse disparity between features in the two-dimensional image and corresponding reference features in a reference image; computing a disparity correction function based on a difference between the first and the second depth coordinates at the matrix of locations, wherein computing the disparity correction function comprises calculating respective local disparity corrections at the matrix of the locations and interpolating and extrapolating from the local disparity corrections to generate the disparity correction function over an entire field of view of the two-dimensional image; correcting the transverse disparity between the two-dimensional image and the reference image using the disparity correction function; and generating a depth map of the target scene based on the corrected transverse disparity.

13 . The method according to claim 12 , wherein directing the modulated optical radiation comprises projecting a pattern of spatially modulated radiation onto the target scene, wherein the reference image comprises a reference pattern, and
wherein deriving the second depth coordinates comprises measuring the transverse disparity between the projected pattern captured in the two-dimensional image and the reference pattern.

14 . The method according to claim 12 , wherein directing the modulated optical radiation comprises directing temporally modulated optical radiation toward the target scene, and wherein sensing the respective times of flight comprises measuring the respective times of flight responsively to a temporal modulation of the reflected photons.

15 . The method according to claim 14 , wherein the temporally modulated radiation comprises a sequenc

Assignees

• Apple Inc

Inventors

• Yosub Shay
• Avraham Assaf
• Nawasra Jawad
• Pokrass Jonathan
• Laifenfeld Moshe
• Ben Zadok Nir
• Gilboa Niv
• Kaitz Tal
• Akerman Ronen
• Levanon Naveh

Classifications

• G06T7/55

  from multiple images · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06T7/521

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G01S7/4865

  Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487) · CPC title