Image restoration method and device

Patent metadata

Field	Value
Publication number	US-12073534-B2
Application number	US-202318296717-A
Country	US
Kind code	B2
Filing date	Apr 6, 2023
Priority date	Jul 24, 2020
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

An image restoration device obtains input data including input image information for each viewpoint, and generates an output image from warped image information generated by warping the input image information using a global transformation information of each viewpoint and disparity information of each viewpoint, using an image restoration model.

First claim

What is claimed is:

1. A processor-implemented method comprising:
capturing a plurality of viewpoint images for a plurality of viewpoints;
generating input data comprising respective input image information for each of the plurality of viewpoints using a feature extraction model;
estimating a respective global transformation parameter of each of the plurality of viewpoints based on the generated input data;
estimating respective disparity information of each of the plurality of viewpoints based on the generated input data;
generating respective warped image information for each of the plurality of viewpoints by warping the respective input image information using the respective global transformation parameter and the respective disparity information; and
generating, using a machine-learning based image restoration model, an output image from the generated respective warped image information for each of the plurality of viewpoints.

2. The method of claim 1 , wherein the capturing comprises:
capturing the plurality of viewpoint images through a plurality of lenses arranged at different positions, and
wherein the generating input data comprises:
obtaining the respective input image information for each of the plurality of viewpoints from the captured plurality of viewpoint images.

3. The method of claim 2 , wherein the obtaining the respective input image information for each of the plurality of viewpoints comprises:
extracting, as the respective input image information, a respective input feature map from each of the plurality of viewpoint images using the feature extraction model.

4. The method of claim 2 , wherein the generating the respective warped image information comprises:
generating a warped feature map by warping a feature map extracted from each of the plurality of viewpoint images.

5. The method of claim 2 , wherein a resolution of the output image is greater than a respective resolution of each of the plurality of viewpoint images.

6. The method of claim 1 , wherein the generating the respective warped image information for each of the plurality of viewpoints comprises:
generating respective transformed image information by transforming the respective input image information into a pixel coordinate system of target image information corresponding to a target viewpoint, using the respective global transformation parameter; and
generating the respective warped image information by correcting a disparity of the generated respective transformed image information with respect to the target image information, using the respective disparity information.

7. The method of claim 6 , wherein the transforming using the respective global transformation parameter comprises:
warping all pixels of the respective input image information to the pixel coordinate system of the target image information, using a single depth corresponding to a reference disparity.

8. The method of claim 7 , wherein the warping comprises:
obtaining a coordinate in the respective input image information corresponding to a position in the pixel coordinate system of the target image information, using the respective global transformation parameter;
obtaining a pixel value of the obtained coordinate in the respective input image information; and
setting a pixel value of the position in the respective transformed image information equal to the obtained pixel value.

9. The method of claim 1 , wherein the estimating the respective global transformation parameter comprises:
obtaining information from the input data through a global pooling operation; and
obtaining the respective global transformation parameter based on the information in which a spatial dimension component is removed.

10. The method of claim 9 , wherein the obtaining the respective global transformation parameter comprises:
applying, to the information in which the spatial dimension component is removed, an operation associated with at least one fully-connected layer of a convolutional neural network.

11. The method of claim 1 , wherein the estimating the respective disparity information comprises:
for each pixel of a plurality of pixels in the respective viewpoint, estimating respective pixel disparity information by performing at least one convolution filtering on feature data extracted from the input data.

12. The method of claim 11 , wherein the estimating the respective pixel disparity information comprises:
obtaining the respective pixel disparity information with a resolution identical to a resolution of the input data.

13. The method of claim 1 , wherein the image restoration model comprises a neural network including at least one convolutional layer that applies convolution filtering to the input data.

14. The method of claim 1 , wherein the generating the output image comprises:
generating image information realigned by a single viewpoint by performing a pixel shuffle on pixels included in the respective warped image information; and
generating the output image having a target resolution by applying the image restoration model to the generated realigned image information.

15. The method of claim 1 , wherein the input data comprises a plurality of pixels, and
wherein the generating of the output image comprises generating the output image without sensing a depth to a respective target point corresponding to each of the plurality of pixels.

16. The method of claim 1 , wherein the obtaining the input data comprises:
capturing a multi-lens image comprising the plurality of viewpoint images by an image sensor comprising a multi-lens array; and
generating the input data from the captured multi-lens image.

17. The method of claim 1 , wherein the obtaining the input data comprises:
capturing the plurality of viewpoint images, each of the plurality of viewpoint images being captured by a respective image sensor of a plurality of image sensors; and
generating the input data from the captured plurality of viewpoint images.

18. A non-transitory computer-readable storage medium storing instructions that are executable by a processor to perform the method of claim 1 .

19. A mobile device comprising:
a camera configured to capture a plurality of viewpoint images for a plurality of viewpoints,
a memory configured to store therein a machine-learning based image restoration model; and
a processor configured to:
generate input data comprising respective input image information for each of the plurality of viewpoints using a feature extraction model
estimate a respective global transformation parameter of each of the plurality of viewpoints based on the generated input data,
estimate respective disparity information of each of the plurality of viewpoints based on the generated input data,
generate respective warped image information for each of the plurality of viewpoints by warping the respective input image information using the respective global transformation parameter and the respective disparity information, and
generate, using the image restoration model, an output image from the generated respective warped image information for each of the plurality of viewpoints.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Seho
• Nam Dong Kyung
• Lee Seok

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T5/60 Primary

  using machine learning, e.g. neural networks · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T7/557

  from light fields, e.g. from plenoptic cameras · CPC title