Upscaling image data

Patent metadata

Field	Value
Publication number	US-12505504-B2
Application number	US-202318104238-A
Country	US
Kind code	B2
Filing date	Jan 31, 2023
Priority date	Aug 22, 2022
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

Image data representing one or more images at a first resolution is received at a first artificial neural network (ANN). The image data is processed using the first ANN to generate upscaled image data representing the one or more images at a second, higher resolution. The first ANN is trained to perform image upscaling and is trained using first training image data representing one or more training images at the first resolution, the first training image data being at a first level of quality. The first ANN is also trained using features of a second ANN, wherein the second ANN is trained to perform image upscaling and is trained using second training image data representing one or more training images at the first resolution, the second training image data being at a second level of quality, higher than the first level of quality.

First claim

What is claimed is:

1 . A computer-implemented method of processing image data, the method comprising:
receiving, at a first artificial neural network, ANN, image data representing one or more images at a first resolution; processing the received image data using the first ANN to generate upscaled image data representing the one or more images at a second resolution, higher than the first resolution; and outputting the upscaled image data from the first ANN, wherein the first ANN is trained to perform image upscaling and is trained using: first training image data representing one or more training images at the first resolution, the first training image data being at a first level of quality; and a feature vector comprising features generated by one or more layers of a second ANN, wherein the second ANN is trained to perform image upscaling and is trained using second training image data representing one or more training images at the first resolution, the second training image data being at a second level of quality, higher than the first level of quality.

2 . The method according to claim 1 , wherein the feature vector comprises an intermediate feature vector comprising features generated by one or more intermediate layers of the second ANN.

3 . The method according to claim 1 , wherein the feature vector is used as a target for training the first ANN.

4 . The method according to claim 1 ,
wherein the first ANN is trained using a feature regularization loss function configured to determine a difference between a feature vector comprising features generated by one or more layers of the first ANN and the feature vector generated by the second ANN, and wherein the first ANN is trained by adjusting the first ANN to reduce the difference as determined by the feature regularization loss function.

5 . The method according to claim 4 , wherein the feature regularization loss function is operable to determine at least one of: an L1-norm loss, a divergence loss, and an adversarial loss, between the feature vector generated by the first ANN and the feature vector generated by the second ANN.

6 . The method according to claim 1 , wherein the first ANN and the second ANN have a same architecture.

7 . The method according to claim 1 , wherein the first ANN and the second ANN comprise a same number of layers and/or parameters, or the first ANN comprises more layers and/or parameters than the second ANN.

8 . The method according to claim 1 , wherein the image data representing the one or more images at the first resolution received at the first ANN is at the first level of quality.

9 . The method according to claim 1 , wherein the first training image data and the second training image data represent a same one or more images at different levels of quality.

10 . The method according to claim 1 , wherein the first training image data is generated by corrupting the second training image data.

11 . The method according to claim 10 , wherein corrupting the second training image data is based on an expected type and/or amount of corruption associated with the image data representing the one or more images received at the first ANN.

12 . The method according to claim 10 , wherein corrupting the second training image data comprises applying noise to the second training image data.

13 . The method according to claim 10 , wherein corrupting the second training image data comprises compressing the second training image data.

14 . The method according to claim 1 , wherein the first ANN and the second ANN are trained simultaneously.

15 . The method according to claim 1 , wherein the first ANN is trained after the second ANN has been trained.

16 . The method according to claim 1 , wherein the first ANN is trained by minimizing losses between upscaled image data, generated by the first ANN using the first training image data and representing the one or more training images at the second resolution, and ground truth image data representing the one or more training images at the second resolution.

17 . The method according to claim 1 , wherein the first ANN is trained using a third ANN configured to distinguish between features of the first ANN and features of the second ANN.

18 . A computer-implemented method of configuring an artificial neural network, ANN, to perform image upscaling, the method comprising:
receiving, at a first ANN, first image data representing one or more training images at a first resolution, the first image data being at a first level of quality; receiving at the first ANN, data derived from features of a second ANN, the second ANN having been trained to generate upscaled image data at a second, higher resolution, the second ANN having been trained using second image data representing one or more training images at the first resolution, the second image data being at a second level of quality, higher than the first level of quality; and using the first image data and the data derived from the features of the second ANN to train the first ANN to perform image upscaling.

19 . A computing device comprising:
a memory comprising computer-executable instructions; a processor configured to execute the computer-executable instructions and cause the computing device to perform a method of processing image data, the method comprising: receiving, at a first artificial neural network, ANN, image data representing one or more images at a first resolution; processing the received image data using the first ANN to generate upscaled image data representing the one or more images at a second resolution, higher than the first resolution; and outputting the upscaled image data from the first ANN, wherein the first ANN is trained to perform image upscaling and is trained using: first training image data representing one or more training images at the first resolution, the first training image data being at a first level of quality; and a feature vector comprising features generated by one or more layers of a second ANN, wherein the second ANN is trained to perform image upscaling and is trained using second training image data representing one or more training images at the first resolution, the second training image data being at a second level of quality, higher than the first level of quality.

20 . A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to perform a method of processing image data, the method comprising:
receiving, at a first artificial neural network, ANN, image data representing one or more images at a first resolution; processing the received image data using the first ANN to generate upscaled image data representing the one or more images at a second resolution, higher than the first resolution; and outputting the upscaled image data from the first ANN, wherein the first ANN is trained to perform image upscaling and is trained using: first training image data representing one or more training images at the first resolution, the first training image data being at a first level of quality; and a feature vector comprising features generated by one or more layers of a second ANN, wherein the second ANN is trained to perform image upscaling and is trained using second training image data representing one or more training images at the first resolution, the second training image data being at a second level of quality, higher than the first level of

Assignees

• Sony Interactive Entertainment Europe Ltd

Inventors

• Khan Muhammad Umar Karim
• Bhunia Ayan
• Chadha Aaron
• Andreopoulos Ioannis

Classifications

• G06N3/096

  Transfer learning · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title