Method and apparatus with image processing for processing depth images

Patent metadata

Field	Value
Publication number	US-12450696-B2
Application number	US-202117383346-A
Country	US
Kind code	B2
Filing date	Jul 22, 2021
Priority date	Dec 15, 2020
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

A method with image processing includes: generating a first surface normal image comprising surface normal vectors corresponding to pixels of a first depth image; and applying the first depth image and the first surface normal image to a first neural network, and acquiring a second depth image by changing the first depth image using the first neural network. The first neural network generates the second depth image to have an improved quality compared to the first depth image, based on an embedding vector that comprises a feature of the first depth image and a feature of the first surface normal image.

First claim

What is claimed is:

1. A processor-implemented image processing method, comprising:
generating a first surface normal image comprising surface normal vectors corresponding to pixels of a first depth image; and
generating, using a first neural network, a second depth image by applying the first depth image and the first surface normal image as input to the first neural network, where the first neural network generates the second depth image by changing the first depth image based on an embedding vector that comprises a feature of the first depth image and a feature of the first surface normal image such that the second depth image has an improved quality compared to the first depth images,
wherein the first neural network removes noise included in the first depth image and increases a sharpness of the first depth image.

2. The method of claim 1 , wherein the first depth image comprises a time-of-flight (TOF) image.

3. The method of claim 1 , further comprising:
generating a second surface normal image comprising surface normal vectors corresponding to pixels corresponding to the second depth image; and
applying the second depth image and the second surface normal image to a second neural network, and acquiring a third depth image by changing the second depth image using the second neural network,
wherein the second neural network generates the third depth image having a degraded quality compared to the second depth image, based on an embedding vector that comprises a feature of the second depth image and a feature of the second surface normal image.

4. The method of claim 3 , further comprising:
generating training data by pairing the second depth image and the third depth image.

5. The method of claim 3 , wherein the first neural network and the second neural network are trained based on a cycle-generative adversarial network (GAN) training method.

6. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

7. A training method, comprising:
acquiring a second depth image of a second type by applying a first depth image of a first type and a first surface normal image corresponding to the first depth image to a first neural network;
acquiring a third depth image of the first type by applying the acquired second depth image and a second surface normal image corresponding to the second depth image to a second neural network;
acquiring a first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network;
acquiring a first cycle consistency loss related to a difference between the third depth image and the first depth image; and
training the first neural network based on the first GAN loss and the first cycle consistency loss, wherein the trained first neural network generates an output depth image by changing an input depth image based on an embedding vector that comprises a feature of the input depth image and a feature of a corresponding surface normal image so that the output depth image has an improved quality compared to the input depth image,
wherein the corresponding surface normal image is generated corresponding to pixels of the input depth image, and
wherein the first neural network removes noise included in the input depth image and increases a sharpness of the input depth image.

8. The training method of claim 7 , wherein the training of the first neural network comprises:
acquiring a fifth depth image of the first type by applying a fourth depth image of the second type and a fourth surface normal image corresponding to the fourth depth image to the second neural network;
acquiring a sixth depth image of the second type by applying the acquired fifth depth image and a fifth surface normal image corresponding to the fifth depth image to the first neural network;
acquiring a second GAN loss related to whether the fifth depth image is an image generated by the second neural network;
acquiring a second cycle consistency loss related to a difference between the sixth depth image and the fourth depth image; and
training the first neural network and the second neural network based on the first GAN loss, the second GAN loss, the first cycle consistency loss, and the second cycle consistency loss.

9. The training method of claim 8 , further comprising:
acquiring a first identity (ID) loss related to a difference between the first depth image and a depth image acquired by applying the first depth image and the first surface normal image to the second neural network;
acquiring a second ID loss related to a difference between the fourth depth image and a depth image acquired by applying the fourth depth image and the fourth surface normal image to the first neural network; and
training the first neural network and the second neural network further based on the first ID loss and the second ID loss.

10. The training method of claim 8 , further comprising:
training the first neural network and the second neural network further based on a first range preserving loss for limiting a difference between the first depth image and the second depth image, and a second range preserving loss for limiting a difference between the fourth depth image and the fifth depth image.

11. The training method of claim 8 , further comprising:
training the second neural network based on a loss for discriminating a difference between the fourth depth image and the fifth depth image and a difference between the fourth depth image and a real depth image of the first type corresponding to the fourth depth image.

12. The training method of claim 7 , wherein
the first neural network comprises a pre-trained neural network trained to generate a depth image of the second type by changing a depth image of the first type in a first direction, and
the second neural network comprises a pre-trained neural network trained to generate a depth image of the first type by changing a depth image of the second type in a second direction opposite to the first direction.

13. The training method of claim 7 , wherein a depth image of the first type comprises a depth image with noise,
wherein a depth image of the second type comprises a depth image that does not contain noise,
wherein the first neural network is configured to generate a depth image having noise removed by removing noise from an input depth image, and
wherein the second neural network is configured to generate a depth image having noise added by adding noise to another input depth image.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 7 .

15. An apparatus with image processing, comprising:
at least one processor;
at least one memory storing instructions, which when executed by the at least one processor configure the at least one processor to:
generate a first surface normal image comprising surface normal vectors corresponding to pixels of a first depth image; and
generate, using a first neural network, a second depth image by applying the first depth image and the first surface normal image as input to the first neural network, where the first neural network generates the second depth image by changing the first depth image based on an embedding vector that comprises a feature of the first depth image and a feature of the first surface normal image such that the second depth image has an improved quality compared to the first depth image,
wherein the first neural network is

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Heewon
• Rhee Seon Min
• Kim Jihye
• Han Seungju

Classifications

• G06T5/73

  Deblurring; Sharpening · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title