Method for denoising time-of-flight range images

Patent metadata

Field	Value
Publication number	US-9805294-B2
Application number	US-201514620610-A
Country	US
Kind code	B2
Filing date	Feb 12, 2015
Priority date	Feb 12, 2015
Publication date	Oct 31, 2017
Grant date	Oct 31, 2017

Abstract

A method for denoising a range image acquired by a time-of-flight (ToF) camera by first determining locations of edges, and a confidence value of each pixel, and based on the locations of the edges, determining geodesic distances of neighboring pixels. Based on the confidence values, reliabilities of the neighboring pixels are determined and scene dependent noise is reduced using a filter.

First claim

We claim:

1. A method for denoising an image, wherein the image is a time-of-flight (ToF) range image, wherein the image includes a 2D grid of pixels, comprising steps of:
determining locations of edges, and a confidence value of each pixel;
determining, geodesic distances of neighboring pixels, wherein the geodesic distances are determined using a binary edge map, wherein a geodesic distance between two neighboring pixels is set to a constant when the two neighboring pixels are not on different sides of an edge of the binary edge map, otherwise, the geodesic distance is set to ∞ infinity;
determining, based on the confidence values, reliabilities of the neighboring pixels; and
reducing scene dependent noise using a filter, wherein the filter uses the geodesic distances and the reliabilities of the neighboring pixels, wherein the ToF range image is acquired by a ToF range camera, and the steps are performed in a processor.

2. The method of claim 1 , further comprising:
calibrating the range image, wherein the calibrating further comprises:
reducing scene independent noise using a pixel-wise calibration using a checkboard calibration pattern, wherein intensities of the checkboard calibration pattern gradually increase from 0% to 50% for darker squares, and from 50% to 100% for lighter squares.

3. The method of claim 1 , wherein the locations of the edges are determined using a neural network.

4. The method of claim 3 , wherein the neural network is learned from a scene dataset including pairs of ToF range images and ground truth range images.

5. The method of claim 4 , wherein the ground truth range images are obtained using at least one structured light sensor.

6. The method of claim 5 , wherein the ToF range camera and the structured light sensor are attached on a robot arm, the ToF range camera and the structured light sensor are calibrated with a robot coordinate system, and the coordinate transformation between the ToF range camera and the structured light sensor is obtained using the robot coordinate system.

7. The method of claim 1 , wherein the confidence values are determined using a neural network.

8. The method of claim 7 , wherein the neural network is learned from a scene dataset including pairs of ToF range images and ground truth range images.

9. The method of claim 8 , wherein the ground truth range images are acquired by a structured light sensor.

10. The method of claim 9 , wherein the structured light sensor and the ToF range camera are attached on a robot arm and moved to different viewpoints to acquire the scene dataset.

11. The method of claim 1 , wherein the ToF range camera is attached on a robot arm and moved to different viewpoints to acquire a plane dataset.

12. A method for denoising an image, the image is a time-of-flight (ToF) range image of pixels, such that the ToF range image is acquired by an input interface connected to a processor, wherein the processor stores the ToF range image in a non-transitory computer readable memory, and embodied in the non-transitory computer readable memory is a program executable by the processor for performing the method, comprising:
determining locations of edges, and a confidence value of each pixel;
determining, based on the locations of the edges, geodesic distances of neighboring pixels, wherein the geodesic distances are determined using a binary edge map, such that the binary edge map is generated based on the locations of the edges, wherein a geodesic distance between two neighboring pixels is set to a constant when the two neighboring pixels are not on different sides of an edge of the binary edge map, otherwise, the geodesic distance is set to ∞ infinity;
determining, based on the confidence values, reliabilities of the neighboring pixels;
reducing scene dependent noise using a filter, such that the filter uses the geodesic distances and the reliabilities of the neighboring pixels, and transforms the image to produce a filtered image; and
outputting the filtered image via an output interface in communication with the processor, wherein noise in the filtered image is less than noise in the image.

13. A system for denoising an image, wherein the image is a time-of-flight (ToF) range image of pixels, comprising:
a non-transitory computer readable memory that includes the image stored therein;
a ToF range camera;
an output interface; and
a processor connected to the non-transitory computer readable memory, wherein the non-transitory computer readable memory includes embodied thereon a program executable by the processor to:
acquire, by the processor, the stored ToF range image, wherein the ToF range image is generated by the ToF range camera;
determine locations of edges, and a confidence value of each pixel;
determine, geodesic distances of neighboring pixels, wherein the geodesic distances are determined using a binary edge map, such that the binary edge map is generated based on the locations of the edges, such that a geodesic distance between two neighboring pixels is set to a constant when the two neighboring pixels are not on different sides of an edge of the binary edge map, otherwise, the geodesic distance is set to ∞ infinity;
determine, based on the confidence values, reliabilities of the neighboring pixels;
reduce scene dependent noise using a filter, such that the filter uses the geodesic distances and the reliabilities of the neighboring pixels, and transforms the image to produce a filtered image; and
output the filtered image via the output interface in communication with the processor, wherein noise in the filtered image is less than noise in the image.

14. The method of claim 13 , wherein the confidence values are determined using a neural network.

15. The method of claim 14 , wherein the neural network is learned from a scene dataset including pairs of ToF range images and ground truth range images, such that the ground truth range images are acquired by at least one structured light sensor.

16. The method of claim 15 , wherein the ToF range camera and the structured light sensor are attached on a robot arm, the ToF range camera and the structured light sensor are calibrated with a robot coordinate system, and the coordinate transformation between the ToF range camera and the structured light sensor is obtained using the robot coordinate system.

17. A method for denoising an image, wherein the image is a time-of-flight (ToF) range image of pixels, comprising steps of:
determining locations of edges, and a confidence value of each pixel;
determining, based on the locations of the edges, geodesic distances of neighboring pixels;
determining, based on the confidence values, reliabilities of the neighboring pixels; and
reducing scene dependent noise using a filter, wherein the filter uses the geodesic distances and the reliabilities of the neighboring pixels, such that the filter is

R
^

⁡

(
p
)

=

∑

q
∈

N
⁡

(
p

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Liu Ming-Yu
• Taguchi Yuichi
• Son Kilho

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G06K9/52

  Physics · mapped topic