Image generation using one or more neural networks

Patent metadata

Field	Value
Publication number	US-2022108417-A1
Application number	US-202017061041-A
Country	US
Kind code	A1
Filing date	Oct 1, 2020
Priority date	Oct 1, 2020
Publication date	Apr 7, 2022
Grant date	—

Abstract

Apparatuses, systems, and techniques are presented to generate images. In at least one embodiment, one or more neural networks are used to generate one or more images based, at least in part, upon speech input received from one or more users.

First claim

What is claimed is:

1 . A processor, comprising:
one or more circuits to use one or more neural networks to generate one or more images based, at least in part, upon speech input received from one or more users.

2 . The processor of claim 1 , wherein the one or more circuits are further to receive the speech input detected as uttered by the one or more users, and determine one or more descriptive features corresponding to the speech input.

3 . The processor of claim 2 , wherein the one or more neural networks include a first neural network to infer a semantic segmentation mask based at least in part upon the one or more descriptive features.

4 . The processor of claim 3 , wherein the one or more neural networks include a second neural network to generate the one or more images based at least in part upon the semantic segmentation mask.

5 . The processor of claim 4 , wherein the second neural network is further to generate the one or more images based further upon one or more style vectors determined from the speech input, the one or more style vectors corresponding to a visual aspect of one or more objects represented in the semantic segmentation mask.

6 . The processor of claim 1 , wherein the one or more circuits are further to generate one or more updated images based, at least in part, upon additional speech input received from the one or more users.

7 . A system comprising:
one or more processors to use one or more neural networks to generate one or more images based, at least in part, upon speech input received from one or more users.

8 . The system of claim 7 , wherein the one or more processors are further to receive the speech input detected as uttered by the one or more users, and determine one or more descriptive features corresponding to the speech input.

9 . The system of claim 8 , wherein the one or more neural networks include a first neural network to infer a semantic segmentation mask based at least in part upon the one or more descriptive features.

10 . The system of claim 9 , wherein the one or more neural networks include a second neural network to generate the one or more images based at least in part upon the semantic segmentation mask.

11 . The system of claim 10 , wherein the second neural network is further to generate the one or more images based further upon one or more style vectors determined from the speech input, the one or more style vectors corresponding to a visual aspect of one or more objects represented in the semantic segmentation mask.

12 . The system of claim 7 , wherein the one or more processors are further to generate one or more updated images based, at least in part, upon additional speech input received from the one or more users.

13 . A method comprising:
using one or more neural networks to generate one or more images based, at least in part, upon speech input received from one or more users.

14 . The method of claim 13 , further comprising:
receiving the speech input detected as uttered by the one or more users; and determining one or more descriptive features corresponding to the speech input.

15 . The method of claim 14 , wherein the one or more neural networks include a first neural network to infer a semantic segmentation mask based at least in part upon the one or more descriptive features.

16 . The method of claim 15 , wherein the one or more neural networks include a second neural network to generate the one or more images based at least in part upon the semantic segmentation mask.

17 . The method of claim 16 , wherein the second neural network is further to generate the one or more images based further upon one or more style vectors determined from the speech input, the one or more style vectors corresponding to a visual aspect of one or more objects represented in the semantic segmentation mask.

18 . The method of claim 13 , further comprising:
using the one or more neural networks to generate one or more updated images based, at least in part, upon additional speech input received from the one or more users.

19 . A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
use one or more neural networks to generate one or more images based, at least in part, upon speech input received from one or more users.

20 . The machine-readable medium of claim 19 , wherein the instructions if performed further cause the one or more processors to:
receive the speech input detected as uttered by the one or more users; and determine one or more descriptive features corresponding to the speech input.

21 . The machine-readable medium of claim 20 , wherein the one or more neural networks include a first neural network to infer a semantic segmentation mask based at least in part upon the one or more descriptive features.

22 . The machine-readable medium of claim 21 , wherein the one or more neural networks include a second neural network to generate the one or more images based at least in part upon the semantic segmentation mask.

23 . The machine-readable medium of claim 22 , wherein the second neural network is further to generate the one or more images based further upon one or more style vectors determined from the speech input, the one or more style vectors corresponding to a visual aspect of one or more obj ects represented in the semantic segmentation mask.

24 . The machine-readable medium of claim 19 , wherein the instructions if performed further cause the one or more processors to generate one or more updated images based, at least in part, upon additional speech input received from the one or more users.

25 . An image generation system, comprising:
one or more processors to use one or more neural networks to generate one or more images based, at least in part, upon speech input received from one or more users; and memory for storing network parameters for the one or more neural networks.

26 . The image generation system of claim 25 , wherein the one or more processors are further to receive the speech input detected, as uttered by the one or more users, and determine one or more descriptive features corresponding to this speech input.

27 . The image generation system of claim 26 , wherein the one or more neural networks include a first neural network to infer a semantic segmentation mask based at least in part upon the one or more descriptive features.

28 . The image generation system of claim 27 , wherein the one or more neural networks include a second neural network to generate the one or more images based at least in part upon the semantic segmentation mask.

29 . The image generation system of claim 28 , wherein the second neural network is further to generate the one or more images based further upon one or more style vectors determined from the speech input, the one or more style vectors corresponding to a visual aspect of one or more obj ects represented in the semantic segmentation mask.

30 . The image generation system of claim 25 , wherein the one or more processors are further to generate one or more updated images based, at least in part, upon additional speech input received from the one or more users.

Assignees

• Nvidia Corp

Inventors

• Liu Ming-Yu
• Huang Xun

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title