Method and apparatus with image processing

Patent metadata

Field	Value
Publication number	US-12198362-B2
Application number	US-202217572908-A
Country	US
Kind code	B2
Filing date	Jan 11, 2022
Priority date	Jul 9, 2021
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

A method and apparatus with image processing are disclosed. The method includes determining a real part image, an imaginary part image, and an offset image based on input images that are dependent on infrared rays of different phases, removing noise from each of the real part image and the imaginary part image using the offset image as a noise removal guide, and generating a depth image based on an improved real part image and an improved imaginary part image corresponding to respective results of the removing.

First claim

What is claimed is:

1. A processor-implemented method, comprising:
determining a real part image, an imaginary part image, and an offset image based on input images, where the input images are dependent on infrared rays of different phases;
removing noise from each of the real part image and the imaginary part image using the offset image as a noise removal guide; and
generating a depth image based on an improved real part image and an improved imaginary part image corresponding to respective results of the removing.

2. The method of claim 1 , further comprising sensing the infrared rays and generating the input images using the sensed infrared rays.

3. The method of claim 2 , further comprising transmitting the infrared rays,
wherein the sensing of the infrared rays includes sensing reflections of the transmitted infrared rays, and the generating of the input images uses the sensed reflections of the transmitted infrared rays.

4. The method of claim 1 , wherein the determining of the real part image, the imaginary part image, and the offset image comprises:
determining the real part image based on a difference in a first image pair among the input images;
determining the imaginary part image based on a difference in a second image pair among the input images different from the first image pair; and
determining the offset image based on a sum of the input images.

5. The method of claim 1 , wherein the removing of the noise comprises:
determining a weighted sum coefficient of a non-local mean (NLM) filter from the offset image; and
determining the improved real part image and the improved imaginary part image by calculating a weighted sum for each of the real part image and the imaginary part image based on the weighted sum coefficient.

6. The method of claim 5 , wherein the determining of the weighted sum coefficient comprises:
determining a weight of a central pixel of a first scan patch of the offset image based on a similarity between a first target patch of the offset image and the first scan patch of the offset image; and
determining a first weighted sum coefficient of the central pixel of the first target patch based on the weight of the central pixel of the first scan patch.

7. The method of claim 1 , wherein the generating of the depth image comprises:
determining a phase image based on the improved real part image and the improved imaginary part image; and
generating the depth image by performing phase wrapping based on the phase image.

8. The method of claim 1 , wherein the removing of the noise comprises:
generating a temporary filtering result by removing noise from the real part image using the offset image as the noise removal guide;
determining a residual image corresponding to a difference between the temporary filtering result and the real part image;
generating an improved residual image by removing noise from the residual image using the offset image as the noise removal guide; and
determining the improved real part image by adding the temporary filtering result and the improved residual image.

9. The method of claim 1 , wherein the removing of the noise comprises:
performing recursive twicing regularization comprising a plurality of noise removal stages of performing the noise removal based on the offset image.

10. The method of claim 9 , wherein, as noise removal on the offset image is gradually performed through the noise removal stages, a different version of the improved offset image is used in each of the noise removal stages.

11. The method of claim 9 , wherein the performing of the recursive twicing regularization comprises:
determining a first improved real part image, a first improved imaginary part image, and a first improved offset image by performing twicing regularization on the real part image, the imaginary part image, and the offset image, respectively, based on the offset image in a first noise removal stage of the noise removal stages; and
determining a second improved real part image, a second improved imaginary part image, and a second improved offset image by performing twicing regularization on the real part image, the imaginary part image, and the offset image, respectively, based on the improved first offset image in a second noise removal stage of the noise removal stages.

12. The method of claim 11 ,
wherein a final improved real part image and a final improved imaginary part image are determined through a final noise removal stage of the noise removal stages, and
wherein the generating of the depth image comprises generating the depth image based on the final improved real part image and the final improved imaginary part image.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1 .

14. An apparatus, comprising:
a processor configured to:
determine a real part image, an imaginary part image, and an offset image based on input images, where the input images are dependent on infrared rays of different phases;
remove noise from each of the real part image and the imaginary part image using the offset image as a noise removal guide; and
generate a depth image based on an improved real part image and an improved imaginary part image corresponding to respective results of the removing.

15. The apparatus of claim 14 , further comprising a sensor configured to sense the infrared rays and generate the input images using the sensed infrared rays.

16. The apparatus of claim 15 , wherein the sensor is configured to transmit the infrared rays, sense reflections of the transmitted infrared rays, and generate the input images using the sensed reflections of the transmitted infrared rays.

17. The apparatus of claim 14 , wherein the processor is configured to:
determine the real part image based on a difference in a first image pair among the input images;
determine the imaginary part image based on a difference in a second image pair among the input images different from the first image pair; and
determine the offset image based on a sum of the input images.

18. The apparatus of claim 14 , wherein the processor is configured to:
determine a weighted sum coefficient of a non-local mean (NLM) filter from the offset image; and
determine the improved real part image and the improved imaginary part image by a weighted sum for each of the real part image and the imaginary part image based on the weighted sum coefficient.

19. The apparatus of claim 18 , wherein, for the determining of the weighted sum coefficient, the processor is configured to:
determine a weight of a central pixel of a first scan patch based on a similarity between a first target patch of the offset image and the first scan patch of the offset image; and
determine a first weighted sum coefficient of the central pixel of the first target patch based on the weight of the central pixel of the first scan patch.

20. The apparatus of claim 14 , wherein the processor is configured to:
perform recursive twicing regularization comprising a plurality of noise removal stages of performing the noise removal based on the offset image.

21. The apparatus of claim 20 , wherein, as noise removal on the offset image is gradually performed through the noise removal stages, a different version of the improved offset image is used in each of the noise removal stages.

22. The apparatus of claim 14 , further comprising a memory storing ins

Assignees

• Samsung Electronics Co Ltd
• Univ Yonsei Iacf

Inventors

• Jee Seunghoon
• Kang Moon Gi
• Lee Sangyoon
• Kim Minsub
• Hong Soonyoung

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T2207/10048

  Infrared image · CPC title

• G01S17/894 Primary

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title