Image restoration using machine learning

Patent metadata

Field	Value
Publication number	US-12530872-B2
Application number	US-202217955846-A
Country	US
Kind code	B2
Filing date	Sep 29, 2022
Priority date	Mar 31, 2020
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

A device comprising an image processor configured to implement: a first machine learning model for performing restoration processing on degraded image data; and a second machine learning model for recognizing areas of an image requiring processing emphasis during the restoration processing, wherein the output of the second machine learning model is an input to the first machine learning model to optimize the restoration processing.

First claim

The invention claimed is:

1 . A device comprising an image processor configured to train an image processing system by training:
a first machine learning model for performing restoration processing on degraded image data; and a second machine learning model for recognizing areas of an image requiring processing emphasis during the restoration processing, wherein an output of the second machine learning model is an input to the first machine learning model to optimize the restoration processing, and wherein training of the first machine learning model comprises:
determining loss data by comparing reconstructed image data generated from the degraded image data with corresponding optimum image data;
combining the loss data with a weight map to form weighted loss data comprising a spatial distribution of the loss data; and
updating the first machine learning model based on the weighted loss data,
wherein the weight map comprises per-pixel weighting values, such that different pixels are assigned different weights based on image characteristics at respective pixel locations, and
wherein combining the loss data with the weight map comprises modulating a contribution of the loss data at each pixel according to the respective per-pixel weighting value in the weight map.

2 . The device according to claim 1 , wherein the training of the first machine learning model further comprises:
receiving training data comprising the degraded image data and the corresponding optimum image data and providing the degraded image data as an initial input to the image processing system; and passing the degraded image data to the first machine learning model configured to generate the reconstructed image data by performing the restoration processing of the degraded image data.

3 . The device according to claim 1 , wherein the second machine learning model is trained by:
receiving the weighted loss data at the second machine learning model; determining by the second machine learning model the spatial distribution of the loss data based on the weighted loss data; and updating the weight map to account for the spatial distribution of the loss data derived from the weighted loss data.

4 . The device according to claim 1 , wherein the second machine learning model is trained to:
identify which spatially distributed regions of a degraded image are more susceptible to degradation based on one or more image features; and generate a weight map for use in performing restoration processing on the degraded image such that a greater weighting is applied to the identified regions.

5 . A method of training an image processing system comprising a first machine learning model, the method comprising training the first machine learning model by:
receiving training data comprising degraded image data and corresponding optimum image data and providing the degraded image data as an input to the image processing system; passing the degraded image data to the first machine learning model configured to create restored image data by restoring the degraded image data; determining loss data by comparing the restored image data to the corresponding optimum image data; combining the loss data with a weight map to form weighted loss data comprising a spatial distribution of the loss data; and updating the first machine learning model based on the weighted loss data, wherein the weight map comprises per-pixel weighting values, such that different pixels are assigned different weights based on image characteristics at respective pixel locations, and wherein combining the loss data with the weight map comprises modulating a contribution of the loss data at each pixel according to the respective per-pixel weighting value in the weight map.

6 . The method according to claim 5 , wherein the image processing system further comprises a second machine learning model and the method further comprises training the second machine learning model by performing an updating process comprising:
receiving the weighted loss data at the second machine learning model; determining by the second machine learning model the spatial distribution of the loss data based on the weighted loss data; and updating the weight map to account for the spatial distribution of the loss data derived from the weighted loss data.

7 . The method according to claim 6 , wherein the training of the image processing system is repeated so as to iteratively update the weight map based on weighted loss data generated from a previous weight map and the first machine learning model in a previous iteration of training the image processing system.

8 . The method according to claim 7 , wherein, in at least some iterations of the training of the image processing system, the training data is different from the training data received in a previous iteration of the training of the image processing system.

9 . The method according to claim 6 , wherein the method further comprises modifying the first machine learning model by combining the first machine learning model with the second machine learning model to create a modified first machine learning model such that the modified first machine learning model is trained to focus on regions of a degraded image that are more susceptible to degradation.

10 . The method according to claim 9 , further comprising:
receiving test data comprising degraded image data and corresponding optimum image data and providing the degraded image data as an input to the modified first machine learning model; creating reconstructed image data by restoration processing of the degraded image data; determining loss data by comparing the reconstructed image data to the corresponding optimum image data; and optimizing the second machine learning model based on the loss data to generate an optimized second machine learning model.

11 . The method according to claim 10 , wherein the weight map is generated by the optimized second machine learning model.

12 . The method of claim 11 , further comprising updating the optimized second machine learning model by implementing an updating process comprising:
receiving weighted loss data at the optimized second machine learning model; determining, by the optimized second machine learning model, the spatial distribution of the loss data based on the weighted loss data; and updating the optimized second machine learning model to generate a weight map to account for the spatial distribution of the loss data derived from the weighted loss data.

13 . The method according to claim 12 , further comprising modifying the modified first machine learning model by combining the updated first machine learning model with the updated optimized second machine learning model to create a second modified first machine learning model such that the second modified first machine learning model is trained to focus on regions of a degraded image that are more susceptible to degradation.

14 . The method according to claim 10 , wherein the restoration processing is a joint denoising and demosaicing processing and the received degraded image data is RAW image data comprising a red, green or blue value for each sampled pixel, such that the first machine learning model is trained to infer a denoised and demosaiced RGB image from the received degraded image data.

15 . A device comprising an image processor, the image processor being configured to train an image processing system comprising a first machine learning model, and the training of the image processing system comprising training the first machine learning model by:
receivin

Assignees

• Huawei Tech Co Ltd

Inventors

• Li Zeju
• Chen Liang
• Slabaugh Gregory
• Liu Liu
• Fu Zhongqian

Classifications

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T3/4015

  Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns · CPC title