AI downscaling apparatus and operating method thereof, and AI upscaling apparatus and operating method thereof

Patent metadata

Field	Value
Publication number	US-12254591-B2
Application number	US-202418638310-A
Country	US
Kind code	B2
Filing date	Apr 17, 2024
Priority date	Apr 14, 2020
Publication date	Mar 18, 2025
Grant date	Mar 18, 2025

Abstract

An artificial intelligence (AI) upscaling apparatus for upscaling a low-resolution image to a high-resolution image includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is configured to: obtain a second image corresponding to a first image, which is downscaled from an original image by an AI downscaling apparatus by using a first deep neural network (DNN); and obtain a third image by upscaling the second image by using a second DNN corresponding to the first DNN, and wherein the second DNN is trained to minimize a difference between a first restored image, which results from applying no pixel movement to an original training image, and second restored images, which result from downscaling, upscaling, and subsequently retranslating one or more translation images obtained by applying pixel movement to the original training image.

First claim

The invention claimed is:

1. An artificial intelligence (AI) upscaling apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory,
wherein the processor is configured to execute the one or more instructions to:
obtain a second image corresponding to a first image which is downscaled from an original image by an AI downscaling apparatus by using a first deep neural network (DNN); and
obtain a third image by upscaling the second image by using a second DNN corresponding to the first DNN, and

wherein the second DNN is trained to minimize a difference between a first restored image for an original training image and second restored images for translation images obtained by applying first pixel movements to the original training image in first directions,
wherein the first restored image is obtained by performing downscaling by the first DNN on the original training image, and performing upscaling by the second DNN on the downscaled image, and
wherein the second restored images are obtained by performing downscaling by the first DNN on the translation images, performing upscaling by the second DNN on the downscaled translation images, and performing retranslation on the upscaled translation images by applying second pixel movements to the upscaled translation images in second directions reverse to the first directions.

2. The AI upscaling apparatus of claim 1 , wherein the second DNN is trained to minimize loss information obtained based on at least one of the original training image, the first restored image for the original training image, or the second restored images for the translation images.

3. The AI upscaling apparatus of claim 2 , wherein the loss information comprises first difference information between the original training image and each of the first restored image and the second restored images.

4. The AI upscaling apparatus of claim 2 , wherein the loss information comprises second difference information between the first restored image and the second restored images.

5. The AI upscaling apparatus of claim 1 , wherein the second DNN receives, as an input, a low-resolution single frame image for a particular time point in the second image and outputs a high-resolution single frame image for the particular time point in the third image.

6. The AI upscaling apparatus of claim 1 , wherein the second DNN comprises a network that is trained jointly with the first DNN and trained based on an image obtained from the first DNN.

7. An artificial intelligence (AI) downscaling apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory,
wherein the processor is configured to execute the one or more instructions to:
obtain a first image that is downscaled from an original image by using a first deep neural network (DNN); and
perform control to transmit the first image to an AI upscaling apparatus through a network, and

wherein the first DNN is trained to minimize a difference between a first restored image for an original training image and second restored images for translation images obtained by applying first pixel movements to the original training image in first directions,
wherein the first restored image is obtained by performing downscaling by the first DNN on the original training image, and performing upscaling by a second DNN on the downscaled image, and
wherein the second restored images are obtained by performing downscaling by the first DNN on the translation images, performing upscaling by the second DNN on the downscaled translation images, and performing retranslation on the upscaled translation images by applying second pixel movements to the upscaled translation images in second directions reverse to the first directions.

8. The AI downscaling apparatus of claim 7 , wherein the first DNN is trained to minimize loss information obtained based on at least one of the original training image, the first restored image for the original training image, or the second restored images for the translation images.

9. The AI downscaling apparatus of claim 8 , wherein the loss information comprises first difference information between the original training image and each of the first restored image and the second restored images.

10. The AI downscaling apparatus of claim 8 , wherein the loss information comprises second difference information between the first restored image and the second restored images.

11. A method of training a first deep neural network (DNN) for downscaling a high-resolution image to a low-resolution image or a second DNN for upscaling a low-resolution image to a high-resolution image, the method comprising:
generating translation images by applying first pixel movement to an original training image in first directions;
obtaining a downscaled image by performing downscaling by the first DNN on the original training image and obtaining downscaled translation images by performing downscaling by the first DNN on the translation images;
obtaining a first restored image by performing upscaling by the second DNN on the downscaled image and obtaining second restored images by performing upscaling by the second DNN on the downscaled translation images and performing retranslation on the upscaled translation images by applying second pixel movements to the upscaled translation images in second directions reverse to the first directions; and
updating at least one of first parameters of the first DNN or second parameters of the second DNN to minimize a difference between the first restored image for the original training image and the second restored images for the translation images by using loss information obtained based on at least one of the original training image, the first restored image for the original training image, or the second restored images for the translation images.

12. The method of claim 11 , wherein the loss information comprises first difference information between the original training image and each of the first restored image and the second restored images.

13. The method of claim 11 , wherein the loss information comprises second difference information between the first restored image and the second restored images.

14. The method of claim 11 , wherein the updating of at least one of the first parameters of the first DNN or the second parameters of the second DNN comprises updating at least one of the first parameters of the first DNN or the second parameters of the second DNN toward minimizing the loss information.

15. The method of claim 11 , wherein the second DNN comprises a network that is trained jointly with the first DNN and trained based on an image obtained from the first DNN.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Wookhyung
• Cho Ilhyun
• Koo Jayoon
• Kim Namuk

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title