Systems and methods for generating voxel-based three-dimensional objects in a virtual space based on natural language processing (NLP) of a user-provided description

Patent metadata

Field	Value
Publication number	US-11568600-B2
Application number	US-202117532457-A
Country	US
Kind code	B2
Filing date	Nov 22, 2021
Priority date	Sep 11, 2020
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

Systems and methods for using natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space are disclosed. Exemplary implementations may: obtain three-dimensional objects using a three-dimensional voxelized format; encode those objects, using a variational autoencoder, into pairs of vectors that are subsequently sampled; decode the sampled vectors; determine loss information for the decoded voxelized three-dimensional objects; use the loss information to train the variational autoencoder; fine-tune a pretrained text-based system; receive user input describing a three-dimensional object; generate a vector from the user input; decode the vector into a voxelized three-dimensional object; present the voxelized three-dimensional object to the user.

First claim

What is claimed is:

1. A system configured to use natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space, the system comprising:
electronic storage configured to electronically store information, wherein the stored information includes a set of three-dimensional objects, wherein the set includes an annotated object that includes a textual description of the annotated object, wherein the annotated object uses a first three-dimensional format; and
one or more hardware processors configured by machine-readable instructions to:
obtain the set of three-dimensional objects, wherein the set of three-dimensional objects includes voxelized three-dimensional objects, wherein the voxelized three-dimensional objects include a first voxelized object based on the annotated object;
encode, using a variational autoencoder that includes an encoder and a decoder, the voxelized three-dimensional objects into pairs of multi-dimensional vectors, wherein individual pairs include a vector of means and a vector of standard deviations;
create sampled multi-dimensional vectors having a particular dimensionality by sampling from the individual pairs of multi-dimensional vectors such that a first sampled multi-dimensional vector is created based on an encoding of the first voxelized object;
decode, using the decoder, the sampled multi-dimensional vectors into decoded voxelized three-dimensional objects;
determine loss information for the decoded voxelized three-dimensional objects by comparing the decoded voxelized three-dimensional objects to corresponding individual voxelized three-dimensional objects;
use the loss information to train the variational autoencoder;
fine-tune a pretrained text-based system to generate multi-dimensional vectors having the particular dimensionality from textual descriptions included in annotated objects in the set of three-dimensional objects, wherein one or more prompts for the pretrained text-based system are based on a combination of the textual description of the annotated object and the first sampled multi-dimensional vector;
receive, from a user, a user input, wherein the user input includes user-provided text describing a three-dimensional object;
generate, using the fine-tuned pretrained text-based system, a user-provided multi-dimensional vector from the user-provided text in the user input;
decode, using the decoder, the user-provided multi-dimensional vector into a first voxelized three-dimensional object; and
present, to the user, the first voxelized three-dimensional object.

2. The system of claim 1 , wherein the set of three-dimensional objects includes one or more objects in one or more three-dimensional formats that fail to support voxels, wherein the one or more three-dimensional formats that fail to support voxels include one or both of a vector-based three-dimensional format and a polygon-based three-dimensional format.

3. The system of claim 1 , wherein the three-dimensional voxelized format that natively supports voxels is selected from one of .OBJ, .JSON, .XML, .SCHEMATIC, .SCAD, .STL, .QB, and .VOX.

4. The system of claim 1 , wherein the loss information includes reconstruction loss and Kullback-Leibler loss, wherein the Kullback-Leibler loss is based on a sum of Kullback-Leibler divergences.

5. The system of claim 1 , wherein the sampled multi-dimensional vectors are decoded using a Generative Adversarial Network that includes a generator and a discriminator.

6. The system of claim 5 , wherein the decoder is used as the generator of the Generative Adversarial Network.

7. The system of claim 6 , wherein the discriminator of the Generative Adversarial Network distinguishes between objects from the set of three-dimensional objects and the decoded voxelized three-dimensional objects as decoded by the decoder.

8. The system of claim 1 , wherein the pretrained text-based system is Generative Pretrained Transformer 3 (GPT-3).

9. The system of claim 1 , wherein the user input is received through a user interface, and wherein the first voxelized three-dimensional object is presented through the user interface.

10. A method to use natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space, the method being implemented in a computer system, the method comprising:
obtaining a set of three-dimensional objects, wherein the set includes an annotated object that includes a textual description of the annotated object, wherein the annotated object uses a first three-dimensional format, wherein the set of three-dimensional objects includes voxelized three-dimensional objects, wherein the voxelized three-dimensional objects include a first voxelized object based on the annotated object;
encoding, using a variational autoencoder that includes an encoder and a decoder, the voxelized three-dimensional objects into pairs of multi-dimensional vectors, wherein individual pairs include a vector of means and a vector of standard deviations;
creating sampled multi-dimensional vectors having a particular dimensionality by sampling from the individual pairs of multi-dimensional vectors such that a first sampled multi-dimensional vector is created based on an encoding of the first voxelized object;
decoding, using the decoder, the sampled multi-dimensional vectors into decoded voxelized three-dimensional objects;
determining loss information by comparing the decoded voxelized three-dimensional objects to corresponding individual voxelized three-dimensional objects;
using the loss information to train the variational autoencoder, including the decoder;
fine-tuning a pretrained text-based system to generate multi-dimensional vectors having the particular dimensionality from textual descriptions included in annotated objects in the set of three-dimensional objects, wherein one or more prompts for the pretrained text-based system are based on a combination of the textual description of the annotated object and the first sampled multi-dimensional vector;
receiving, from a user, a user input, wherein the user input includes user-provided text describing a three-dimensional object;
generating, using the fine-tuned pretrained text-based system, a user-provided multi-dimensional vector from the user-provided text in the user input;
decoding, using the decoder, the user-provided multi-dimensional vector into a first voxelized three-dimensional object; and
presenting, to the user, the first voxelized three-dimensional object.

11. The method of claim 10 , wherein the set of three-dimensional objects includes one or more objects in one or more three-dimensional formats that fail to support voxels, wherein the one or more three-dimensional formats that fail to support voxels include one or both of a vector-based three-dimensional format and a polygon-based three-dimensional format.

12. The method of claim 10 , wherein the three-dimensional voxelized format that natively supports voxels is selected from one of .OBJ, .JSON, .XML, .SCHEMATIC, .SCAD, .STL, .QB, and .VOX.

13. The method of claim 10 , wherein the loss information includes reconstruction loss and Kullback-Leibler loss, wherein the Kullback-Leibler loss is based on a sum of Kullback-Leibler divergences.

14. The method of claim 10 , wherein the sampled multi-dimensional vectors are decoded using a Generative Adversarial Network that includes a generator and a discriminator.

15. The method of claim 14 , wherein the decoder is used as the generator of the Generative Adversarial Network.

16. The method of claim 15 , wherein the discriminator of

Assignees

• Mythical Inc

Inventors

• Thacker Cameron
• Linden John

Classifications

• G06T9/00

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• G06T17/10 Primary

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06F40/14

  Tree-structured documents (parsing G06F40/205; validation G06F40/226) · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title