Image upscaling

Patent metadata

Field	Value
Publication number	US-9996902-B2
Application number	US-201615000670-A
Country	US
Kind code	B2
Filing date	Jan 19, 2016
Priority date	Jan 19, 2016
Publication date	Jun 12, 2018
Grant date	Jun 12, 2018

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for upscaling an image. One of the methods includes upscaling a low resolution image, creating first pixel subsets of the first upscaled image, creating second pixel subsets of a high resolution image, determining, for each subset in the pixel subsets, a value of a property of the pixel subset, determining, for each subset in the pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property, and determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image.

First claim

What is claimed is:

1. A system comprising:
a data processing apparatus; and
a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:
upscaling a low resolution image with a first resolution that depicts particular content to a second resolution greater than the first resolution to create a first upscaled image;
creating first pixel subsets of the first upscaled image, each subset in the first pixel subsets having a first size, each pixel in the first upscaled image included in a subset in the first pixel subsets;
creating second pixel subsets of a high resolution image with a third resolution, greater than the first resolution, that depicts the particular content, each subset in the second pixel subsets having a second size, each pixel in the high resolution image included in a subset in the second pixel subsets, a quantity of the first pixel subsets being the same as a quantity of the second pixel subsets and each of the first pixel subsets corresponding to a respective one of the second pixel subsets, wherein the high resolution image is a separate image from the first upscaled image;
determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a value of a property of the corresponding pixel subset using a) the corresponding pixel subset and b) at least one pixel subset adjacent to the corresponding pixel subset that includes one or more pixels that are not included in the corresponding pixel subset;
determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a group of subsets to which the corresponding pixel subset belongs using the value of the property of the corresponding pixel subset, each pixel subset included in one and only one group; and
determining, for each of the groups of subsets, a filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create a final pixel subset that approximates the corresponding second pixel subset using the first pixel subset, a combination of all of the final pixel subsets representing a second upscaled image.

2. The system of claim 1 , the operations comprising:
receiving the low resolution image prior to upscaling the low resolution image; and
receiving the high resolution image prior to creating the second pixel subsets of the high resolution image.

3. The system of claim 1 , wherein upscaling the low resolution image with the first resolution that depicts particular content to the second resolution greater than the first resolution to create the first upscaled image comprises upscaling the low resolution image to the third resolution of the high resolution image.

4. The system of claim 1 , wherein:
the first size is the same size as the second size; and
creating the second pixel subsets of the high resolution image, each subset in the second pixel subsets having the second size comprises creating the second pixel subsets of the high resolution image, each subset in the second pixel subsets having the first size.

5. The system of claim 1 , wherein:
creating the first pixel subsets of the first upscaled image comprises creating first non-overlapping pixel subsets of the first upscaled image; and
creating second pixel subsets of the high resolution image with the third resolution comprises creating second non-overlapping pixel subsets of the high resolution image.

6. The system of claim 1 , wherein:
creating the first pixel subsets of the first upscaled image comprises creating first overlapping pixel subsets of the first upscaled image; and
creating second pixel subsets of the high resolution image with the third resolution comprises creating second overlapping pixel subsets of the high resolution image.

7. The system of claim 1 , wherein determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset comprises determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, an angle of an edge in the corresponding pixel subset.

8. The system of claim 1 , wherein determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset comprises determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, a derivative of an edge in the corresponding pixel subset.

9. The system of claim 1 , wherein determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset comprises determining, for each subset in the first pixel subsets or for each subset in the second pixel subsets, the value of the property of the corresponding pixel subset using the corresponding first pixel subset and all pixel subsets adjacent to the corresponding pixel subset that each include one or more pixels that are not included in the corresponding pixel subset.

10. The system of claim 1 , wherein:
determining, for each of the groups of subsets, the filter comprises determining, during a training process and for each of the groups of subsets, the filter; and
the operations comprise compressing, during the training process, the low resolution image prior to upscaling the low resolution image.

11. The system of claim 1 , wherein:
determining, for each of the groups of subsets, the filter comprises determining, during a training process and for each of the groups of subsets, the filter; and
the operations comprise sharpening, during the training process, the high resolution image prior to creating the second pixel subsets of the high resolution image.

12. The system of claim 1 , the operations comprising:
determining, for each of the groups of subsets, a second filter to apply to each of the final pixel subsets in the group to create a second final pixel subset that approximates the corresponding second pixel subset using the final pixel subset, a combination of all of the second final pixel subsets representing a third upscaled image.

13. The system of claim 1 , wherein determining, for each of the groups of subsets, the filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create the final pixel subset that approximates the corresponding second pixel subset using the first pixel subset comprises:
determining, for each of the groups of subsets, a horizontal filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group; and
determining, for each of the groups of subsets, a vertical filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group.

14. The system of claim 1 , wherein determining, for each of the groups of subsets, the filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group to create the final pixel subset that approximates the corresponding second pixel subset using the first pixel subset comprises determining, for each of the groups of subsets, a unique filter to apply to each of the first pixel subsets that correspond to the pixel subsets in the group that is different from the filters for the other groups.

15. The system of claim 1 , wherein the data processing apparatus automatically upscales

Assignees

• Google Llc

Inventors

• Milanfar Peyman
• Romano Yaniv

Classifications

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2210/36

  Level of detail · CPC title

• G06T3/403

  Edge-driven scaling; Edge-based scaling · CPC title