Upsampling an image using one or more neural networks

Patent metadata

Field	Value
Publication number	US-12573000-B2
Application number	US-202017066282-A
Country	US
Kind code	B2
Filing date	Oct 8, 2020
Priority date	Oct 8, 2020
Publication date	Mar 10, 2026
Grant date	Mar 10, 2026

Abstract

Apparatuses, systems, and techniques are presented to generate images. In at least one embodiment, one or more neural networks are used to generate one or more images using one or more pixel weights determined based, at least in part, on one or more sub-pixel offset values.

First claim

What is claimed is:

1 . A processor, comprising:
one or more circuits to use one or more neural networks to determine one or more blending weights based, at least in part, on one or more sub-pixel offset values determined from a jitter-aware upsampling process, and to generate one or more upsampled images using the one or more blending weights.

2 . The processor of claim 1 , wherein a jitter-aware upsampling process is used to upsample one or more rendered images.

3 . The processor of claim 2 , wherein the one or more rendered images correspond to an image sequence, and wherein the one or more blending weights are to be used to perform a weighted blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images of the image sequence.

4 . The processor of claim 3 , wherein the one or more blending weights include a first blending weight for a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations, wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations in the one or more rendered images as determined using the one or more sub-pixel offset values.

5 . The processor of claim 4 , wherein the one or more blending weights include a second blending weight for a second blending of all pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations to generate one or more output images in the image sequence.

6 . The processor of claim 5 , wherein the first blending weight and the second blending weight are determined by the one or more neural networks, and wherein the first blending weight is higher than the second blending weight.

7 . A system comprising:
one or more processors to use one or more neural networks to determine one or more blending weights based, at least in part, on one or more sub-pixel offset values determined from a jitter-aware upsampling process, and to generate one or more upsampled images using the one or more blending weights.

8 . The system of claim 7 , wherein a jitter-aware upsampling process is used to upsample one or more rendered images.

9 . The system of claim 8 , wherein the one or more rendered images correspond to an image sequence, and wherein the one or more blending weights are to be used to perform a weighted blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images of the image sequence.

10 . The system of claim 9 , wherein the one or more blending weights include a first blending weight for a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations, wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations in the one or more rendered images as determined using the one or more sub-pixel offset values.

11 . The system of claim 10 , wherein the one or more blending weights include a second blending weight for a second blending of all pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations to generate one or more output images in the image sequence.

12 . The system of claim 11 , wherein the first blending weight and the second blending weight are determined by the one or more neural networks, and wherein the first blending weight is higher than the second blending weight.

13 . A method comprising:
using one or more neural networks to determine one or more blending weights based, at least in part, on one or more sub-pixel offset values determined from a jitter-aware upsampling process, and to generate one or more upsampled images using the one or more blending weights.

14 . The method of claim 13 , wherein a jitter-aware upsampling process is used to upsample one or more rendered images.

15 . The method of claim 14 , wherein the one or more rendered images correspond to an image sequence, and wherein the one or more blending weights are to be used to perform a weighted blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images of the image sequence.

16 . The method of claim 15 , wherein the one or more blending weights include a first blending weight for a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations, wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations in the one or more rendered images as determined using the one or more sub-pixel offset values.

17 . The method of claim 16 , wherein the one or more blending weights include a second blending weight for a second blending of all pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations to generate one or more output images in the image sequence.

18 . The method of claim 17 , wherein the first blending weight and the second blending weight are determined by the one or more neural networks, and wherein the first blending weight is higher than the second blending weight.

19 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use one or more neural networks to determine one or more blending weights based, at least in part, on one or more sub-pixel offset values determined from a jitter-aware upsampling process, and to generate one or more upsampled images using the one or more blending weights.

20 . The non-transitory machine-readable medium of claim 19 , wherein a jitter-aware upsampling process is used to upsample one or more rendered images.

21 . The non-transitory machine-readable medium of claim 20 , wherein the one or more rendered images correspond to an image sequence, and wherein the one or more blending weights are to be used to perform a weighted blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images of the image sequence.

22 . The non-transitory machine-readable medium of claim 21 , wherein the one or more blending weights include a first blending weight for a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations, wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations in the one or more rendered images as determined using the one or more sub-pixel offset values.

23 . The non-transitory machine-readable medium of claim 22 , wherein the one or more blending weights include a second blending weight for a second blending of all pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations to generate one or more output images in the image sequence.

24 . The non-transitory machine-readable medium of cla

Assignees

• Nvidia Corp

Inventors

• Liu Shiqiu
• Pottorff Robert
• Liu Guilin
• Sapra Karan
• Barker Jon
• Tarjan David
• Janis Pekka
• Fagerholm Edvard
• Yang Lei
• Shih Kevin
• Salvi Marco
• Roman Timo
• Tao Andrew
• Catanzaro Bryan

Classifications

• H04N23/64

  Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• H04N25/48

  Increasing resolution by shifting the sensor relative to the scene · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title