AI downscaling apparatus and operating method thereof, and AI upscaling apparatus and operating method thereof

Patent metadata

Field	Value
Publication number	US-11989852-B2
Application number	US-202117312276-A
Country	US
Kind code	B2
Filing date	Jan 11, 2021
Priority date	Apr 14, 2020
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

An artificial intelligence (AI) upscaling apparatus for upscaling a low-resolution image to a high-resolution image includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is configured to: obtain a second image corresponding to a first image, which is downscaled from an original image by an AI downscaling apparatus by using a first deep neural network (DNN); and obtain a third image by upscaling the second image by using a second DNN corresponding to the first DNN, and wherein the second DNN is trained to minimize a difference between a first restored image, which results from applying no pixel movement to an original training image, and second restored images, which result from downscaling, upscaling, and subsequently retranslating one or more translation images obtained by applying pixel movement to the original training image.

First claim

The invention claimed is:

1. An artificial intelligence (AI) upscaling apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory,
wherein the processor is configured to execute the one or more instructions to:
obtain a second image corresponding to a first image which is downscaled from an original image by an AI downscaling apparatus by using a first deep neural network (DNN); and
obtain a third image by upscaling the second image by using a second DNN corresponding to the first DNN, and
wherein the second DNN is trained to minimize a difference between a first restored image for an original training image and second restored images for translation images,
wherein the first restored image is obtained by performing downscaling by the first DNN on an image obtained by applying no pixel movement to the original training image, and performing upscaling by the second DNN on the downscaled image,
wherein the second restored images are obtained by performing downscaling by the first DNN on the translation images obtained by applying pixel movement to the original training image, performing upscaling by the second DNN on the downscaled translation images, and performing retranslation on the upscaled translation images,
wherein the translation images comprise a first translation image generated by applying pixel movement to the original training image in a first direction, and a second translation image generated by applying pixel movement to the original training image in a second direction vertical to the first direction,
wherein the upscaled translation images comprise a first upscaled translation image corresponding to the first translation image and a second upscaled translation image corresponding to the second translation image, and
wherein the retranslation comprises pixel movement applied to the first upscaled translation image in a direction reverse to the first direction of the pixel movement applied to the original training image, and pixel movement applied to the second upscaled translation image in a direction reverse to the second direction of the pixel movement applied to the original training image.

2. The AI upscaling apparatus of claim 1 , wherein the second DNN is trained to minimize loss information obtained based on at least one of the original training image, the first restored image for the original training image, or the second restored images for the translation images.

3. The AI upscaling apparatus of claim 2 , wherein the loss information comprises first difference information between the original training image and each of the first restored image and the second restored images.

4. The AI upscaling apparatus of claim 2 , wherein the loss information comprises second difference information between the first restored image and the second restored images.

5. The AI upscaling apparatus of claim 1 , wherein the second DNN receives, as an input, a low-resolution single frame image for a particular time point in the second image and outputs a high-resolution single frame image for the particular time point in the third image.

6. The AI upscaling apparatus of claim 1 , wherein the second DNN comprises a network that is trained jointly with the first DNN and trained based on an image obtained from the first DNN.

7. An artificial intelligence (AI) downscaling apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory,
wherein the processor is configured to execute the one or more instructions to:
obtain a first image that is downscaled from an original image by using a first deep neural network (DNN); and
perform control to transmit the first image to an AI upscaling apparatus through a network, and
wherein the first DNN is trained to minimize a difference between a first restored image for an original training image and second restored images for translation images,
wherein the first restored image is obtained by performing downscaling by the first DNN on an image obtained by applying no pixel movement to the original training image, and performing upscaling by a second DNN on the downscaled image,
wherein the second restored images are obtained by performing downscaling by the first DNN on the translation images that are obtained by applying pixel movement to the original training image, performing upscaling by the second DNN on the downscaled translation images, and performing retranslation on the upscaled translation images,
wherein the translation images comprise a first translation image generated by applying pixel movement to the original training image in a first direction, and a second translation image generated by applying pixel movement to the original training image in a second direction vertical to the first direction,
wherein the upscaled translation images comprise a first upscaled translation image corresponding to the first translation image and a second upscaled translation image corresponding to the second translation image, and
wherein the retranslation comprises pixel movement applied to the first upscaled translation image in a direction reverse to the first direction of the pixel movement applied to the original training image, and pixel movement applied to the second upscaled translation image in a direction reverse to the second direction of the pixel movement applied to the original training image.

8. The AI downscaling apparatus of claim 7 , wherein the first DNN is trained to minimize loss information obtained based on at least one of the original training image, the first restored image for the original training image, or the second restored images for the translation images.

9. The AI downscaling apparatus of claim 8 , wherein the loss information comprises first difference information between the original training image and each of the first restored image and the second restored images.

10. The AI downscaling apparatus of claim 8 , wherein the loss information comprises second difference information between the first restored image and the second restored images.

11. A method of training a first deep neural network (DNN) for downscaling a high-resolution image to a low-resolution image or a second DNN for upscaling a low-resolution image to a high-resolution image, the method comprising:
generating translation images by applying pixel movement to an original training image;
obtaining a plurality of low-resolution images corresponding to the original training image and the translation images by performing a downscaling operation with the first DNN on the original training image and the translation images, a low-resolution image corresponding to the original training image among the plurality of low-resolution images being an image obtained by applying no pixel movement to the original training image;
obtaining a plurality of high-resolution images corresponding to the plurality of low-resolution images by performing an upscaling operation with the second DNN on each of the plurality of low-resolution images;
obtaining second restored images by applying retranslation to high-resolution images corresponding to the translation images from among the plurality of high-resolution images; and
updating at least one of first parameters of the first DNN or second parameters of the second DNN to minimize a difference between a first restored image for the original training image and the second restored images for the translation images by using loss information obtained based on at least one of the original training image, the first restored image for the orig

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Wookhyung
• Cho Ilhyun
• Koo Jayoon
• Kim Namuk

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title