Enhanced depth mapping using visual inertial odometry

Patent metadata

Field	Value
Publication number	US-12153140-B2
Application number	US-201917268961-A
Country	US
Kind code	B2
Filing date	Sep 2, 2019
Priority date	Sep 25, 2018
Publication date	Nov 26, 2024
Grant date	Nov 26, 2024

Abstract

Imaging apparatus ( 22 ) includes a radiation source ( 40 ), which emits pulsed beams ( 42 ) of optical radiation toward a target scene ( 24 ). An array ( 52 ) of sensing elements outputs signals indicative of respective times of incidence of photons on the sensing elements. Objective optics ( 54 ) form a first image of the target scene on the array of sensing elements. An image sensor ( 64 ) captures a second image of the target scene. Processing and control circuitry ( 56, 58 ) is configured to process the second image so as to detect a relative motion between at least one object in the target scene and the apparatus, and which is configured to construct, responsively to the signals from the array, histograms of the times of incidence of the photons on the sensing elements and to adjust the histograms responsively to the detected relative motion, and to generate a depth map of the target scene based on the adjusted histograms.

First claim

The invention claimed is:

1. An imaging apparatus, comprising:
a radiation source, which is configured to emit pulsed beams of optical radiation toward a target scene;
an array of sensing elements, configured to output signals indicative of respective times of incidence of photons on the sensing elements;
objective optics configured to form a first image of the target scene on the array of sensing elements;
an image sensor, which is configured to capture a second image of the target scene; and
processing and control circuitry, which is configured to process the second image so as to detect a relative motion between at least one object in the target scene and the apparatus, and which is configured to construct, in response to the signals from the array, histograms of the times of incidence of the photons on the sensing elements and to adjust the histograms in response to the detected relative motion, and to generate a depth map of the target scene based on the adjusted histograms.

2. The imaging apparatus according to claim 1 , wherein the relative motion is due to a movement of the apparatus, and wherein the processing and control circuitry is configured to filter the histograms to compensate for the movement of the apparatus.

3. The imaging apparatus according to claim 2 , and comprising an inertial sensor, which is configured to sense the movement of the apparatus and to output an indication of the movement, wherein the processing and control circuitry is configured to apply the indication output by the inertial sensor in conjunction with processing the second image in detecting the movement of the apparatus.

4. The imaging apparatus according to claim 1 , wherein the processing and control circuitry is configured, upon detecting an absence of the relative motion between the target scene and the apparatus, to extend an exposure time over which the histograms are accumulated.

5. The imaging apparatus according to claim 1 , wherein the relative motion comprises a movement of an object in the target scene, and wherein the processing and control circuitry is configured to filter the histograms to compensate for the movement of the object.

6. The imaging apparatus according to claim 5 , wherein the processing and control circuitry is configured to process the second image so as to extract a trajectory of the movement of the object, and to correct the histograms for the sensing elements onto which the trajectory is imaged by the objective optics.

7. The imaging apparatus according to claim 1 , wherein the processing and control circuitry is configured to identify edges among the histograms, and to apply the identified edges in detecting the relative motion.

8. The imaging apparatus according to claim 1 , wherein the second image is a color image.

9. The imaging apparatus according to claim 1 , wherein the sensing elements comprise single-photon avalanche diodes (SPADs).

10. An imaging apparatus, comprising:
a radiation source, which is configured to emit pulsed beams of optical radiation toward a target scene;
an array of sensing elements, configured to output signals indicative of respective times of incidence of photons on the sensing elements;
objective optics configured to form a first image of the target scene on the array of sensing elements;
an image sensor, which is configured to capture a second image of the target scene; and
processing and control circuitry, which is configured to process the second image so as to estimate a depth range of at least one object in the target scene, and which is configured to construct, in response to the signals from the array, histograms of the times of incidence of the photons on the sensing elements while gating a time range one or more of the histograms in response to the estimated depth range, and to generate a depth map of the target scene based on the adjusted histograms.

11. A method for imaging, comprising:
directing pulsed beams of optical radiation toward a target scene;
imaging the target scene onto an array of sensing elements in an imaging device;
receiving signals from the sensing elements that are indicative of respective times of incidence of photons on the sensing elements;
capturing an image of the target scene;
processing the captured image so as to detect a relative motion between at least one object in the target scene and the imaging device;
constructing, in response to the signals from the array, histograms of the times of incidence of the photons on the sensing elements;
adjusting the histograms in response to the detected relative motion; and
generating a depth map of the target scene based on the adjusted histograms.

12. The method according to claim 11 , wherein the relative motion is due to a movement of the imaging device, and wherein adjusting the histograms comprises filtering the histograms to compensate for the movement of the imaging device.

13. The method according to claim 12 , and comprising sensing the movement of the imaging device using an inertial sensor, which outputs an indication of the movement, wherein filtering the histograms comprises applying the indication output by the inertial sensor in conjunction with processing the image in detecting the movement of the imaging device.

14. The method according to claim 11 , wherein constructing the histograms comprises, upon detecting an absence of the relative motion between the target scene and the imaging device, extending an exposure time over which the histograms are accumulated.

15. The method according to claim 11 , wherein the relative motion comprises a movement of an object in the target scene, and wherein adjusting the histograms comprises filtering the histograms to compensate for the movement of the object.

16. The method according to claim 15 , wherein processing the image comprises extracting a trajectory of the movement of the object, and filtering the histograms comprises correcting the histograms for the sensing elements onto which the trajectory is imaged.

17. The method according to claim 11 , wherein adjusting the histograms comprises identifying edges among the histograms, and applying the identified edges in detecting the relative motion.

18. The method according to claim 11 , wherein the image is a color image.

19. The method according to claim 11 , wherein the sensing elements comprise single-photon avalanche diodes (SPADs).

20. A method for depth mapping, comprising:
directing pulsed beams of optical radiation toward a target scene and receiving signals indicative of respective times of incidence of photons reflected from the target scene on an array of sensing elements in an imaging device;
constructing, in response to the signals from the array that are accumulated over a selected exposure time, histograms of the times of incidence of the photons on the sensing elements;
capturing an image of the target scene;
processing the captured image so as to detect a relative motion between an object in the target scene and the imaging device;
receiving from an inertial sensor an indication of movement of the imaging device;
upon detecting that the imaging device and the target scene are stationary, increasing the exposure time over which the histograms are accumulated;
upon detecting that the imaging device has moved, filtering the histograms to correct for the movement; and
upon detecting that the object has moved, correcting the histograms for the motion of the object.

Assignees

• Apple Inc

Inventors

• Silver David
• Hirsh Eitan
• Laifenfeld Moshe
• Kaitz Tal

Classifications

• G01S17/58

  Velocity or trajectory determination systems; Sense-of-movement determination systems · CPC title

• G01S17/10

  using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32) · CPC title

• G01S7/4863

  Detector arrays, e.g. charge-transfer gates · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title