Method for restoring images and video using self-supervised learning

Patent metadata

Field	Value
Publication number	US-11461881-B2
Application number	US-202017105135-A
Country	US
Kind code	B2
Filing date	Nov 25, 2020
Priority date	Nov 25, 2020
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

A method for processing images comprising: capturing a plurality of degraded images of a first real-world environment with a first sensor; processing each degraded image with a first, untrained convolutional neural network, via a Deep Image Prior approach, to obtain a plurality of clean images, wherein each clean image corresponds to a degraded image; pairing each clean image with its corresponding degraded image to create a plurality of degraded/clean image pairs; training, via a supervised learning approach, a machine learning model to learn a function for converting degraded images into restored images based on the plurality of degraded/clean image pairs; capturing a second plurality of degraded images of a second real-world environment; and using the trained machine learning model to convert the second plurality of degraded images into restored images based on the learned function.

First claim

We claim:

1. A method for processing images comprising:
capturing a plurality of degraded images of a first real-world environment with a first sensor;
processing each degraded image with a first, untrained convolutional neural network, via a Deep Image Prior approach, to obtain a plurality of clean images, wherein each degraded image is processed based only on itself without reference to any other image such that its corresponding clean image is obtained without any other input or training, wherein each clean image corresponds to a single degraded image;
pairing each clean image with its corresponding degraded image to create a plurality of degraded/clean image pairs;
training, via a supervised learning approach, a machine learning model to learn a function for converting degraded images into restored images based on the plurality of degraded/clean image pairs;
capturing a second plurality of degraded images of a second real-world environment; and
using the trained machine learning model to convert the second plurality of degraded images into restored images based on the learned function.

2. The method of claim 1 , wherein the step of capturing the second plurality of degraded images is performed by the first sensor.

3. The method of claim 2 , wherein the first sensor is a video camera and wherein the trained machine learning model is a second convolutional neural network configure to convert live-streaming video images from the first sensor into a restored video stream in real time based on the learned function.

4. The method of claim 2 , wherein the first sensor is selected from a group consisting of a still-image camera, a sonar imager, a thermal imager, and a multispectral sensor.

5. The method of claim 1 , wherein the step of capturing the second plurality of degraded images is performed by a second sensor.

6. The method of claim 5 , wherein the second sensor is a video camera and wherein the trained machine learning model is configured to convert live-streaming video images from the second sensor into a restored video stream in real time based on the learned function.

7. The method of claim 6 , wherein the first and second sensors are thermal imagers.

8. The method of claim 6 , wherein the first and second sensors are multispectral sensors.

9. The method of claim 1 , further comprising continually repeating the steps of claim 1 , wherein all the degraded/clean image pairs created in accordance with claim 1 are used to train the machine learning model so as to continually adapt to sensor degradations and changing environmental conditions.

10. The method of claim 1 , wherein the second real-world environment is at a same location as the first real-world environment but at a later point in time.

11. The method of claim 1 , further comprising repeating the steps of claim 1 at different times of day and in different seasons, wherein all the degraded/clean image pairs created in accordance with claim 1 are used to train the machine learning model.

12. The method of claim 1 , wherein the Deep Image Prior approach is characterized by:
θ*=argminE( f θ ( z ); x 0 ), x*=f θ* ( z )

where f is a learned function, θ is a random initialization of network parameters, θ* is a local minimizer obtained using gradient descent, E(f θ (z); x 0 ) is a task-specific data term, x 0 is a given degraded image; x* is a corresponding restored image, and z is a randomly-initialized three-dimensional tensor.

13. The method of claim 1 , wherein the trained machine learning model is a convolutional neural network configured to convert the second plurality of degraded images into restored images based on the learned function in real time.

14. A system for processing images in real time comprising:
a first sensor configured to capture a plurality of degraded images of a real-world environment;
a first computer processor communicatively coupled to the first sensor, wherein the first computer processor comprises a convolutional neural network configured to process each of the degraded images, via a Deep Image Prior approach, to obtain a plurality of clean images, wherein each clean image corresponds to a single degraded image, such that each clean image is paired with its corresponding degraded image to create a plurality of degraded/clean image pairs;
a second computer processor comprising a machine learning model that has been trained, via a supervised learning approach, to learn a function for converting degraded images into restored images based on the plurality of degraded/clean image pairs; and
wherein the first sensor is communicatively coupled to the second computer processor and configured to capture a second plurality of degraded images of a second real-world environment, wherein the machine learning model is configured to convert the second plurality of degraded images into restored images based on the learned function.

15. The system of claim 14 , wherein the machine learning model of the second computer processor is a convolutional neural network.

16. The system of claim 14 , wherein the Deep Image Prior approach comprises:
initializing the convolutional neural network with random weights to enhance a given degraded image;
using the convolutional neural network's structure as an image prior; and
performing denoising, super-resolution, and inpainting operations on the given degraded image based only on the image prior without any other input or training.

17. The system of claim 16 , wherein the Deep Prior Approach comprises:
initializing the convolutional neural network with a combination of learned weights and random weights to enhance a given degraded image;
using the convolutional neural network's structure as an image prior; and
performing denoising, super-resolution, and inpainting operations on the given degraded image based only on the image prior without any other input or training.

18. The system of claim 14 , wherein the Deep Image Prior approach is characterized by:
θ*=argminE( f θ ( z ); x 0 ), x*=f θ* ( z )

where f is a learned function, θ is a random initialization of network parameters, θ* is a local minimizer obtained using gradient descent, E(f θ (z); x 0 ) is a task-specific data term, x 0 is a given degraded image; x* is a corresponding restored image, and z is a randomly-initialized three-dimensional tensor.

19. The system of claim 14 , wherein the machine learning model is trained, via the supervised learning approach, to learn the function for converting degraded images into restored images based on the plurality of degraded/clean image pairs and a plurality of curated pairs of clean and degraded images.

20. The system of claim 14 , wherein the machine learning model is configured to convert the second plurality of degraded images into restored images based on the learned function in real time.

Assignees

• Us Gov As Represented By The Secretary Of The Navy
• Us Navy

Inventors

• Parameswaran Shibin
• Jaszewski Martin Thomas

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title