Image resolution enhancement

Patent metadata

Field	Value
Publication number	US-10438322-B2
Application number	US-201715607321-A
Country	US
Kind code	B2
Filing date	May 26, 2017
Priority date	May 26, 2017
Publication date	Oct 8, 2019
Grant date	Oct 8, 2019

Abstract

Resolution enhancement techniques are described. An apparatus may receive first image data at a first resolution, and second image data at a resolution less than the first resolution. The second image data may be scaled to the first resolution and compared to the first image data. Application of a neural network may scale the first image data to a resolution higher than the first resolution. The application of the neural network may incorporate signals based on the scaled second image data. The signals may include information obtained by comparing the scaled second image data to the resolution of the first image data.

First claim

What is claimed:

1. An apparatus, comprising:
a first camera configured to obtain first image data of a first resolution;
operatively coupled to the first camera, an image processing system including a resolution-expanding neural network configured to
scale the first image data to a resolution greater than the first resolution, wherein the neural network is trained by using the neural network to scale, to the first resolution, second image data acquired by a second camera at a second resolution lower than the first resolution, with comparison of the first image data versus the second image data scaled to the first resolution and subsequent refinement of neural-network parameters.

2. The apparatus of claim 1 , wherein the first image data and the second image data are each scaled by an equivalent scaling factor.

3. The apparatus of claim 1 , wherein the first camera has a maximum resolution greater than a maximum resolution of the second camera.

4. The apparatus of claim 1 , wherein the image-processing system is configured to:
apply the neural network to the second image data;
calculate an error value based on a comparison of the scaled second image data to the first image data; and
train the neural network based on the error value.

5. The apparatus of claim 1 , wherein the neural-network parameters include a plurality of weights associated with the neural network.

6. The apparatus of claim 1 ,
wherein the first image data and the second image data are acquired contemporaneously.

7. The apparatus of claim 1 , wherein the first image data and the second image data correspond to a common region of an image frame acquired by each of the first and second cameras contemporaneously.

8. A method comprising:
receiving first image data from a first camera at a first resolution; and
scaling the first image data to a resolution higher than the first resolution by application of a non-linear function to the first image data, wherein one or more coefficients of the non-linear function are selected by using the non-linear function to scale, to the first resolution, second image data of a second resolution lower than the first resolution, by comparison of the first image data versus the second image data scaled to the first resolution and subsequent refinement of the one or more coefficients.

9. The method of claim 8 , wherein each of the first image data and the second image data is scaled by a common scaling factor.

10. The method of claim 8 , further comprising:
receiving the second image data from a second camera, wherein the second camera has a maximum resolution less than a maximum resolution of the first camera.

11. The method of claim 8 , further comprising:
scaling the second image data by applying the non-linear function to the second image data, unsealed from the second camera.

12. The method of claim 8 , wherein the non-linear function is a first non-linear function, the method further comprising:
scaling the second image data by applying a second non-linear function trained independently of the first non-linear function.

13. The method of claim 8 , wherein the second image data is obtained from a second camera.

14. The method of claim 13 , wherein the second image data is obtained from the first camera.

15. An apparatus, comprising:
a first camera;
a second camera;
at least one processor; and
computer memory having stored thereon instructions that, when executed by the at least one processor, cause the apparatus to:
obtain first image data from the first camera, the first image data having a first resolution;
obtain second image data from the second camera, the second image data having a second resolution, lower than the first resolution;
use a resolution-expanding neural network to scale the second image data to the first resolution;
train the resolution-expanding neural network based on a comparison of the first image data versus the second image data scaled to the first resolution; and
use the trained resolution-expanding neural network to scale the first image data to a resolution higher than the first resolution.

16. The apparatus of claim 15 , wherein the first image data and the second image data are each scaled by a common scaling factor.

17. The apparatus of claim 15 , wherein the first camera has a maximum resolution greater than the maximum resolution of the second camera.

18. The apparatus of claim 15 , wherein the first image data and the second image data are acquired contemporaneously.

19. The apparatus of claim 15 , wherein the first image data and the second image data correspond to a common region of an image frame acquired by each of the first and second cameras contemporaneously.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lutz Moshe R
• Pradeep Vivek

Classifications

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title

• H04N7/18

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title