Apparatus and method for correcting image regions following upsampling or frame interpolation

Patent metadata

Field	Value
Publication number	US-11620729-B2
Application number	US-202217573392-A
Country	US
Kind code	B2
Filing date	Jan 11, 2022
Priority date	Dec 28, 2018
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

Apparatus and method for correcting image regions following upsampling or frame interpolation. For example, one embodiment of an apparatus comprises a machine-learning engine to evaluate at least a first image in a sequence of images generated by a real-time interactive application, the machine learning engine to responsively use previously learned data to generate an upsampled or interpolated image comprising a plurality of pixel patches. In one embodiment, each pixel patch is associated with a confidence value reflecting how accurately the pixel patch was generated by the machine learning engine. A selective ray tracing engine identifies a first pixel patch to be corrected based a first confidence value corresponding to the first pixel patch being lower than a threshold and performs ray tracing operations on a first portion of the first image to generate a corrected first pixel patch.

First claim

What is claimed is:

1. A method comprising:
evaluating a first pixel patch in a first image to determine a confidence value associated with the first pixel patch, the confidence value determined based on: 1) an evaluation of one or more other images in a sequence of images on which an interpolation or upsampling for generating the first image is based, and/or 2) an analysis motion data associated with the first pixel patch across one or more images in the sequence;
detecting that the confidence value is below a target quality threshold value; and
responsive to the detecting, performing one or more shading operations to generate a second pixel patch with greater shading detail than the first pixel patch.

2. The method of claim 1 , wherein the second pixel patch is to replace the first pixel patch in the first image prior to displaying the first image on a display.

3. The method of claim 1 , wherein the second pixel patch is to be included in a second image prior to displaying the second image on a display.

4. The method of claim 1 , wherein the confidence value is determined by a neural network-based rendering engine.

5. The method of claim 1 , wherein the second pixel patch is computed by a neural network engine.

6. The method of claim 1 , wherein the confidence value comprises a decimal value or a percentage.

7. The method of claim 1 , wherein the one or more shading operations comprise variable rate shading, coarse pixel shading, and/or other adjustable shading techniques.

8. The method of claim 1 , wherein the first pixel patch comprises N×N pixels and the second pixel patch comprises M×M pixels, where M>N.

9. The method of claim 1 , wherein the second pixel patch is of higher image quality than the first pixel patch.

10. A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform operations of:
evaluating a first pixel patch in a first image to determine a confidence value associated with the first pixel patch, the confidence value determined based on: 1) an evaluation of one or more other images in a sequence of images on which an interpolation or upsampling for generating the first image is based, and/or 2) an analysis motion data associated with the first pixel patch across one or more images in the sequence;
detecting that the confidence value is below a target quality threshold value; and
responsive to the detecting, performing one or more shading operations to generate a second pixel patch with greater shading detail than the first pixel patch.

11. The non-transitory machine-readable medium of claim 10 , wherein the second pixel patch is to replace the first pixel patch in the first image prior to displaying the first image on a display.

12. The non-transitory machine-readable medium of claim 10 , wherein the second pixel patch is to be included in a second image prior to displaying the second image on a display.

13. The non-transitory machine-readable medium of claim 10 , wherein the confidence value is determined by a neural network-based rendering engine.

14. The non-transitory machine-readable medium of claim 10 , wherein the second pixel patch is computed by a neural network engine.

15. The non-transitory machine-readable medium of claim 10 , wherein the confidence value comprises a decimal value or a percentage.

16. The non-transitory machine-readable medium of claim 10 , wherein the one or more shading operations comprise variable rate shading, coarse pixel shading, and/or other adjustable shading techniques.

17. The non-transitory machine-readable medium of claim 10 , wherein the first pixel patch comprises N×N pixels and the second pixel patch comprises M×M pixels, where M>N.

18. The non-transitory machine-readable medium of claim 10 , wherein the second pixel patch is of higher image quality than the first pixel patch.

19. An apparatus comprising:
an image upsampler/interpolator to evaluate a first pixel patch in a first image to determine a confidence value associated with the first pixel patch, the confidence value determined based on: 1) an evaluation of one or more other images in a sequence of images on which an interpolation or upsampling for generating the first image is based, and/or 2) an analysis motion data associated with the first pixel patch across one or more images in the sequence; and
a selective rendering engine to detect that the confidence value is below a target quality threshold value and responsively perform one or more shading operations to generate a second pixel patch with greater shading detail than the first pixel patch.

20. The apparatus of claim 19 , wherein the second pixel patch is to replace the first pixel patch in the first image prior to displaying the first image on a display.

21. The apparatus of claim 19 , wherein the second pixel patch is to be included in a second image prior to displaying the second image on a display.

22. The apparatus of claim 19 , wherein the image upsampler/interpolator comprises a neural network-based rendering engine.

23. The apparatus of claim 19 , wherein the confidence value comprises a decimal value or a percentage.

24. The apparatus of claim 19 , wherein the first pixel patch comprises N×N pixels and the second pixel patch comprises M×M pixels, where M>N.

25. The apparatus of claim 19 , wherein the second pixel patch is of higher image quality than the first pixel patch.

Assignees

• Intel Corp

Inventors

• Pohl Daniel

Classifications

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06T1/60

  Memory management · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T15/06

  Ray-tracing · CPC title