Image generation using a neural network

Patent metadata

Field	Value
Publication number	US-12536798-B2
Application number	US-202217701037-A
Country	US
Kind code	B2
Filing date	Mar 22, 2022
Priority date	Feb 8, 2022
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Apparatuses, systems, and techniques to generate an image. In at least one embodiment, one or more neural networks are to generate a second image based, at least in part, on a first image and information indicating zero or more differences between the first and second image.

First claim

What is claimed is:

1 . A processor, comprising:
one or more circuits to:
determine a first image to be a new keyframe based, at least in part, on comparison of a difference between the first image and a current keyframe to a threshold; and
use one or more neural networks to:
generate a first feature map for the first image; and
generate a second feature map of a second image based, at least in part, on the first feature map of the first image and information indicating zero or more differences between the first image and the second image, wherein the first image and the second image are different frames of a video.

2 . The processor of claim 1 , wherein prior to generation of the second feature map, the one or more circuits are further to compress the information indicating zero of more differences using a sparse compression technique based, at least in part, on a number of non-zero values in an array of the information and a number of zero values in the array.

3 . The processor of claim 1 , wherein prior to generation of the second feature map, the one or more circuits are further to determine the second image is a not keyframe based, at least in part, on comparison of a difference between the second image and the first image to the threshold.

4 . The processor of claim 1 , wherein the one or more neural networks are to generate the second feature map of the second image based, at least in part, on a feature map of the differences between the first image and the second image.

5 . The processor of claim 1 , wherein the information indicating the zero or more differences indicates pixel differences between the first image and the second image.

6 . The processor of claim 4 , wherein the one or more neural networks are to generate the second feature map of the second image based, at least in part, on a combination of the first feature map and the feature map of the differences between the first image and the second image.

7 . The processor of claim 1 , wherein the one or more neural networks are to augment the second image with feature map information based, at least in part, on the second feature map.

8 . The processor of claim 1 , wherein the one or more neural networks are to generate a third feature map of a third image based, at least in part, on the first feature map of the first image and information indicating zero or more differences between the first image and the third image, wherein the first image, the second image, and the third image are different frames of a video from a video source.

9 . The processor of claim 1 , wherein the zero or more differences between the first and second images are to be determined using per-pixel subtraction.

10 . The processor of claim 1 , wherein the zero or more differences between the first and second images are to be determined based, at least in part, on applying one or more convolution filters to the first and second images.

11 . A computer-implemented method, comprising:
determining a first image to be a new keyframe based, at least in part, on comparison of a difference between the first image and a current keyframe to a threshold; and
using one or more neural networks to:
generate a first feature map for the first image; and

generate a second feature map of a second image based, at least in part, on the first feature map of the first image and information indicating zero or more differences between the first image and the second image, wherein the first image and the second image are different frames of a video.

12 . The computer-implemented method of claim 11 , further comprising, prior to generation of the second feature map, compressing the information indicating zero of more differences using a sparse compression technique based, at least in part, on a number of non-zero values in an array of the information and a number of zero values in the array.

13 . The computer-implemented method of claim 11 , further comprising, prior to generation of the second feature map, determining the second image is a not keyframe based, at least in part, on comparison of a difference between the second image and the first image to the threshold.

14 . The computer-implemented method of claim 11 , wherein the second feature map of the second image is to be generated based, at least in part, on a feature map of the zero or more differences between the first image and the second image.

15 . The computer-implemented method of claim 11 , wherein the information indicating the zero or more differences indicates pixel differences between the first image and the second image.

16 . The computer-implemented method of claim 11 , further comprising:
compressing the first image prior to using the one or more neural networks to generate the second feature map of the second image.

17 . The computer-implemented method of claim 14 , further comprising:
combining the feature map of the zero or more differences between the first image and the second image with the first feature map to generate a combined feature map as the second feature map.

18 . The computer-implemented method of claim 11 , wherein:
the first feature map is to be used to generate a plurality of feature maps based, at least in part, on the first feature map and information indicating zero or more differences between the first image and respective images of a plurality of other images.

19 . The computer-implemented method of claim 14 , wherein:
a first neural network of the one or more neural networks is to generate the feature map of the zero or more differences between the first image and the second image; and a second neural network of the one or more neural networks is to generate the second feature map based, at least in part, on the first feature map and the feature map of the zero or more differences between the first image and the second image.

20 . The computer-implemented method of claim 11 , wherein the zero or more differences between the first and second images are to be determined based, at least in part, on applying one or more convolution filters to the first and second images.

21 . A computer system, comprising:
one or more processors and memory storing executable instructions that, if performed by the one or more processors, cause the one or more processors to:
determine a first image to be a new keyframe based, at least in part, on comparison of a difference between the first image and a current keyframe to a threshold; and
use one or more neural networks to:
generate a first feature map for the first image; and
generate a second feature map of a second image based, at least in part, on the first feature map of the first image and information indicating zero or more differences between the first image and the second image, wherein the first image and the second image are different frames of a video.

22 . The computer system of claim 21 , wherein the executable instructions cause the one or more processors to, prior to generation of the second feature map, compress the information indicating zero of more differences using a sparse compression technique based, at least in part, on a number of non-zero values in an array of the information and a number of zero values in the array.

23 . The computer system of claim 21 , wherein the executable instructions cause the one or more processors to, prior to generation of the second feature map, determine the second image is a not keyframe based, at

Assignees

• Nvidia Corp

Inventors

• Yu Chong

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V20/46 Primary

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title