System for depth data filtering based on amplitude energy values

Patent metadata

Field	Value
Publication number	US-10242454-B2
Application number	US-201715415724-A
Country	US
Kind code	B2
Filing date	Jan 25, 2017
Priority date	Jan 25, 2017
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

An electronic device includes a time of flight (ToF) camera and one or more processors. The ToF camera captures raw depth images. The processors determine a depth frame and an amplitude frame from the raw depth images. The depth frame comprises an array of pixels, each pixel having a depth value. The amplitude frame comprises an array of pixels, each pixel having an amplitude energy value. The processors determine a first energy threshold value based on the amplitude energy values of the array of pixels of the amplitude frame and determine, for the depth value of a first pixel of the depth frame, a confidence value representing a corresponding validity of a depth represented by the depth value, based on a comparison of the amplitude energy value of a corresponding first pixel of the amplitude frame to the first energy threshold value.

First claim

What is claimed is:

1. A method comprising:
capturing one or more raw depth images using a time-of-flight camera of an electronic device;
determining, by one or more processors of the electronic device, a depth frame from the one or more raw depth images, the depth frame comprising an array of pixels, each pixel of the depth frame having a depth value;
determining, by the one or more processors of the electronic device, an amplitude frame from the one or more raw depth images, the amplitude frame comprising an array of pixels, each pixel of the amplitude frame having an amplitude energy value;
determining, by the one or more processors of the electronic device, a first energy threshold value based on the amplitude energy values of the array of pixels of the amplitude frame;
determining, by the one or more processors of the electronic device, for each of the amplitude energy values of the amplitude frame, a corresponding confidence value representing a corresponding validity of a depth represented by the depth value of a corresponding pixel of the depth frame, based on a comparison of the amplitude energy value of the pixel of the amplitude frame to the first energy threshold value;
filtering the array of pixels of the amplitude frame based on the confidence values; and
identifying spatial features in imagery captured by a visible light camera of the electronic device based on the filtered array of pixels.

2. The method of claim 1 , wherein determining the first energy threshold value based on the amplitude energy values of the array of the pixels of the amplitude frame comprises:
determining an estimated pixel energy value for the amplitude frame based on the amplitude energy values of the amplitude frame; and
setting the first energy threshold value based on the estimated pixel energy value for the amplitude frame.

3. The method of claim 2 , wherein determining the estimated pixel energy value for the amplitude frame based on the amplitude energy values comprises:
determining a sum of the amplitude energy values for the pixels of the amplitude frame; and
determining the estimated pixel energy to be an average amplitude energy value for the sum.

4. The method of claim 2 , wherein determining the estimated pixel energy value for the amplitude frame based on the amplitude energy values comprises:
generating a cumulative distribution function based on the amplitude energy values for the pixels of the amplitude frame; and
determining the estimated pixel energy value to be a value such that a specified percentage of the pixels of the amplitude frame have an amplitude energy value equal or less than the value.

5. The method of claim 2 , wherein the first energy threshold value is a specified percentage of the estimated pixel energy value.

6. The method of claim 2 , wherein the first energy threshold value is less than the estimated pixel energy value.

7. The method of claim 1 , wherein the filtering comprises:
identifying the amplitude energy value of a first pixel of the amplitude frame as invalid when the amplitude energy value of the first pixel of the amplitude frame is less than the first energy threshold value.

8. The method of claim 1 , filtering comprises:
identifying a first pixel of the amplitude frame as valid when the amplitude energy value of the first pixel is greater than or equal to the first energy threshold value.

9. The method of claim 1 , wherein filtering comprises:
identifying the corresponding pixel of the depth frame as invalid for use in depth sensing when the amplitude energy value of a first pixel of the amplitude frame is below the first energy threshold value.

10. The method of claim 1 , further comprising:
setting a second energy threshold value based on the amplitude energy values of the amplitude frame; and
wherein determining the confidence value further comprises:
characterizing the corresponding pixel of the depth frame based on the amplitude energy value of a first pixel of the amplitude frame compared to the second energy threshold value.

11. An electronic device comprising:
a time of flight (ToF) camera to capture one or more raw depth images; and
a processor configured to:
determine a depth frame from the one or more raw depth images, the depth frame comprising an array of pixels, each pixel of the depth frame having a depth value;
determine an amplitude frame from the one or more raw depth images, the amplitude frame comprising an array of pixels, each pixel of the amplitude frame having an amplitude energy value;
determine a first energy threshold value based on the amplitude energy values of the pixels of the amplitude frame; and
determine, for each pixel of the array of pixels of the amplitude frame, a corresponding confidence value representing a corresponding validity of a depth represented by the depth value of a corresponding pixel of the depth frame, based on a comparison of the amplitude energy value of the corresponding pixel of the amplitude frame to the first energy threshold value;
filter the array of pixels of the amplitude frame based on the confidence values; and
identify spatial features in imagery captured by a visible light camera of the electronic device of the electronic device based on the filtered array of pixels.

12. The electronic device of claim 11 wherein the processor is further configured to:
determine an estimated pixel energy value for the amplitude frame based on the amplitude energy values of the amplitude frame; and
set the first energy threshold value based on the estimated pixel energy value for the amplitude frame.

13. The electronic device of claim 12 wherein the processor is further configured to
determine a sum of the amplitude energy values for the pixels of the amplitude frame; and
determine the estimated pixel energy value to be an average amplitude energy value for the sum.

14. The electronic device of claim 12 wherein the processor is further configured to
generate a cumulative distribution function based on the amplitude energy values for the pixels of the amplitude frame; and
determine the estimated pixel energy value to be a value such that a specified percentage of the pixels of the amplitude frame have an amplitude energy value equal or less than the value.

15. The electronic device of claim 11 wherein the processor is configured to filter by identifying the depth value of a pixel of the depth frame as invalid for use in depth sensing when the amplitude energy value of a first pixel of the amplitude frame is below the first energy threshold value.

16. The electronic device of claim 11 , wherein the processor is configured to filter by abstaining from using a corresponding first pixel of the depth frame for depth sensing when the amplitude energy value of a first pixel of the amplitude frame is below the first energy threshold value.

17. The electronic device of claim 11 further comprising:
a display for displaying a rendering of the amplitude frame and imagery captured by a visible light camera of the electronic device.

18. A method for determining a depth of each of one or more spatial features in imagery captured by a visible light camera of an electronic device using one or more processors of the electronic device, the method comprising:
capturing one or more raw depth images using a depth sensor of the electronic device;
determining, by one or more processors of the electronic device, a depth frame comprising pixels from the one or more raw depth images, the pixels of the depth frame having depth values represen

Assignees

• Google Inc
• Google Llc

Inventors

• Schmidt Mirko

Classifications

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G01S17/87

  Combinations of systems using electromagnetic waves other than radio waves · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title