Systems and methods for depth map sampling

Patent metadata

Field	Value
Publication number	US-10282591-B2
Application number	US-201514833573-A
Country	US
Kind code	B2
Filing date	Aug 24, 2015
Priority date	Aug 24, 2015
Publication date	May 7, 2019
Grant date	May 7, 2019

Abstract

An electronic device is described. The electronic device includes a camera configured to capture an image of a scene. The electronic device also includes an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene. The electronic device further includes a memory configured to store the image and the spatial coordinates. The electronic device additionally includes a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values corresponding to at least a subset of the spatial coordinates. The electronic device further includes a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

First claim

What is claimed is:

1. An electronic device, comprising:
a camera configured to capture an image of a scene;
an image segmentation mapper configured to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;
a memory configured to store the image and the spatial coordinates;
a LIDAR (light+radar) unit, the LIDAR unit steerable to selectively obtain depth values of the scene corresponding to the spatial coordinates; and
a depth mapper configured to generate a depth map of the scene based on the depth values and the spatial coordinates.

2. The electronic device of claim 1 , wherein the image segmentation mapper is configured to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

3. The electronic device of claim 1 , wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

4. The electronic device of claim 1 , wherein the sampling point for each image segment corresponds to a centroid of the segment.

5. The electronic device of claim 1 , wherein the image segmentation mapper is configured to provide the sampling point for each image segment to the LIDAR unit and to provide the segments of the image to the depth mapper.

6. The electronic device of claim 1 , wherein the depth mapper is configured to generate the depth map by merging the segments with corresponding depth values.

7. The electronic device of claim 1 , wherein the depth mapper is configured to generate the depth map by populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

8. The electronic device of claim 1 , wherein the number of depth values obtained by the LIDAR unit is configured to be adjusted based on feedback from a prior depth map.

9. A method, comprising:
capturing an image of a scene;
performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;
obtaining, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and
generating a depth map of the scene based on the depth values and the spatial coordinates.

10. The method of claim 9 , wherein performing segmentation of the image comprises performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

11. The method of claim 9 , wherein the LIDAR unit is configured to perform a coarse scan over a region of the scene containing a substantially uniform object within the scene.

12. The method of claim 9 , wherein generating the depth map comprises merging the segments with corresponding depth values.

13. The method of claim 9 , wherein generating the depth map comprises populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

14. The method of claim 9 , further comprising adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

15. An apparatus, comprising:
means for capturing an image of a scene;
means for performing segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;
means for obtaining, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and
means for generating a depth map of the scene based on the depth values and the spatial coordinates.

16. The apparatus of claim 15 , wherein the means for performing segmentation of the image comprise means for performing segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

17. The apparatus of claim 15 , wherein the sampling point for each image segment corresponds to a centroid of the segment.

18. The apparatus of claim 15 , wherein the means for generating the depth map comprise means for merging the segments with corresponding depth values.

19. The apparatus of claim 15 , wherein the means for generating the depth map comprise means for populating each segment with a corresponding depth value obtained by the LIDAR unit at the spatial coordinates.

20. The apparatus of claim 15 , further comprising means for adjusting the number of depth values obtained by the LIDAR unit based on feedback from a prior depth map.

21. A computer-program product, comprising a non-transitory tangible computer-readable medium having instructions thereon, the instructions comprising:
code for causing an electronic device to capture an image of a scene;
code for causing the electronic device to perform segmentation of the image based on image content to generate a plurality of image segments, each of the plurality of image segments associated with spatial coordinates indicative of a location of each segment in the scene, wherein at least a portion of the image segments comprise non-uniform segments having irregular geometry, size, and distribution based on the image content, and wherein the spatial coordinates of each image segment correspond to a sampling point for each image segment;
code for causing the electronic device to obtain, by a LIDAR unit, depth values of the scene corresponding to the spatial coordinates, wherein the LIDAR unit is steerable to selectively obtain the depth values; and
code for causing the electronic device to generate a depth map of the scene based on the depth values and the spatial coordinates.

22. The computer-program product of claim 21 , wherein the code for causing the electronic device to perform segmentation of the image comprises code for causing the electronic device to perform segmentation based on an image complexity, wherein a quantity of segments generated is a function of a determined complexity of the image.

23. The computer-program product of claim 21 , wherein the sampling point for each image segment corresponds to a centroid of the segment.

24. The computer-program product of claim 21 , wherein the code for causing the electr

Assignees

• Qualcomm Inc

Inventors

• Lindner Albrecht Johannes
• Slobodyanyuk Volodimir
• Verrall Stephen Michael
• Atanassov Kalin Mitkov

Classifications

• G01S17/86 Primary

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• H04N13/261

  with monoscopic-to-stereoscopic image conversion · CPC title

• G06T2207/10024

  Color image · CPC title

• G01S17/89

  for mapping or imaging · CPC title