Super-resolution based foveated rendering

Patent metadata

Field	Value
Publication number	US-10643307-B2
Application number	US-201715809344-A
Country	US
Kind code	B2
Filing date	Nov 10, 2017
Priority date	Nov 10, 2017
Publication date	May 5, 2020
Grant date	May 5, 2020

Abstract

An embodiment of a semiconductor package apparatus may include technology to identify a region of interest portion of a first image, and render the region of interest portion with super-resolution. Other embodiments are disclosed and claimed.

First claim

We claim:

1. An electronic processing system comprising:
a graphics processor;
memory communicatively coupled to the graphics processor; and
logic communicatively coupled to the graphics processor to:
identify a region of interest portion of a first image,
provide the region of interest portion of the first image to a super-resolution neural network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image,
up-sample the first image to generate an up-sampled image,
combine the super-resolution enhanced image with the up-sampled image to provide a foveated image, and
train the super-resolution neural network to provide a blended transition between the region of interest portion of the first image and other portions of the first image.

2. The system of claim 1 , wherein the logic is further to:
crop a training image based on the region of interest to generate a cropped image;
down-sample the cropped image to generate a down-sampled image;
up-sample the down-sampled image to generate an up-sampled second image; and
blend the up-sampled second image with the cropped image to generate a target image.

3. The system of claim 2 , wherein the logic is further to:
train the super-resolution neural network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

4. The system of claim 1 , wherein the resolution of the first image is lower than a resolution of a target display device.

5. A semiconductor package apparatus comprising:
one or more substrates; and
logic coupled to the one or more substrates, wherein the logic is at least partly implemented in one or more of configurable logic and fixed-functionality hardware logic, the logic coupled to the one or more substrates to:
identify a region of interest portion of a first image,
provide the region of interest portion of the first image to a super-resolution neural network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image,
up-sample the first image to generate an up-sampled image,
combine the super-resolution enhanced image with the up-sampled image to provide a foveated image, and
train the super-resolution neural network to provide a blended transition between the region of interest portion of the first image and other portions of the first image.

6. The apparatus of claim 5 , wherein the logic is further to:
crop a training image based on the region of interest to generate a cropped image;
down-sample the cropped image to generate a down-sampled image;
up-sample the down-sampled image to generate an up-sampled second image; and
blend the up-sampled second image with the cropped image to generate a target image.

7. The apparatus of claim 6 , wherein the logic is further to:
train the super-resolution neural network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

8. The apparatus of claim 5 , wherein the resolution of the first image is lower than a resolution of a target display device.

9. A method of processing an image, comprising:
cropping a training image to generate a cropped image;
down-sampling the cropped image to generate a down-sampled image;
up-sampling the down-sampled image to generate an up-sampled image;
blending the up-sampled image with the cropped image to generate a target image; and
training a super-resolution network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

10. The method of claim 9 , further comprising:
identifying a region of interest portion of a first image;
providing the region of interest portion of the first image to the trained super-resolution network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image;
up-sampling the first image to generate an up-sampled second image; and
combining the super-resolution enhanced image with the up-sampled second image to provide a foveated image.

11. The method of claim 10 , wherein the resolution of the first image is lower than a resolution of a target display device.

12. The method of claim 9 , wherein the super-resolution network comprises a super-resolution neural network.

13. The method of claim 12 , wherein the super-resolution neural network comprises a super-resolution convolutional neural network.

14. At least one non-transitory computer readable medium, comprising a set of instructions, which when executed by a computing device, cause the computing device to:
crop a training image to generate a cropped image;
down-sample the cropped image to generate a down-sampled image;
up-sample the down-sampled image to generate an up-sampled image;
blend the up-sampled image with the cropped image to generate a target image; and
train a super-resolution network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

15. The at least one non-transitory computer readable medium of claim 14 , comprising a further set of instructions, which when executed by the computing device, cause the computing device to:
identify a region of interest portion of a first image;
provide the region of interest portion of the first image to the trained super-resolution network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image;
up-sample the first image to generate an up-sampled second image; and
combine the super-resolution enhanced image with the up-sampled second image to provide a foveated image.

16. The at least one non-transitory computer readable medium of claim 15 , wherein the resolution of the first image is lower than a resolution of a target display device.

17. The at least one non-transitory computer readable medium of claim 14 , wherein the super-resolution network comprises a super-resolution neural network.

18. The at least one non-transitory computer readable medium of claim 17 , wherein the super-resolution neural network comprises a super-resolution convolutional neural network.

Assignees

• Intel Corp

Inventors

• Chen Fan
• Li Zhengmin

Classifications

• G06T3/4046

  using neural networks · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title