Interactive neural field editing in content generation systems and applications

Patent metadata

Field	Value
Publication number	US-2025111588-A1
Application number	US-202318479261-A
Country	US
Kind code	A1
Filing date	Oct 2, 2023
Priority date	Oct 2, 2023
Publication date	Apr 3, 2025
Grant date	—

Abstract

Systems and methods of the present disclosure include interactive editing for generated three-dimensional (3D) models, such as those represented by neural radiance fields (NeRFs). A 3D model may be presented to a user in which the user may identify one or more localized regions for editing and/or modification. The localized regions may be selected and a corresponding 3D volume for that region may be provided to one or more generative networks, along with a prompt, to generate new content for the localized regions. Each of the original NeRF and the newly generated NeRF for the new content may then be combined into a single NeRF for a combined 3D representation with the original content and the localized modifications.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
receiving an input corresponding to content for a selected portion of a 3D volume of a first neural radiance field (NeRF) corresponding to a scene; generating a second NeRF of the selected portion based, at least, on the input; and generating an output NeRF by blending the first NeRF and the second NeRF.

2 . The computer-implemented method of claim 1 , further comprising:
determining, for a region of the first NeRF, a feature value; determining, for the region of the second NeRF, an overwriting feature value; and replacing, in the output NeRF, the feature value with the overwriting feature value.

3 . The computer-implemented method of claim 1 , wherein the input is a text prompt.

4 . The computer-implemented method of claim 1 , further comprising:
receiving an interaction to the first NeRF, the interaction corresponding to the selected portion; determining, based at least on a parameter of the interaction, a depth for the selected portion; and generating the selected portion using the depth.

5 . The computer-implemented method of claim 1 , wherein the output NeRF is generated by a diffusion model.

6 . The computer-implemented method of claim 1 , wherein the input is associated with a tool used to make the selection.

7 . The computer-implemented method of claim 1 , wherein the selected portion is a two-dimensional selection based on a first camera view, further comprising:
generating one or more rays from a virtual camera associated with the first camera view; identifying one or more cells interacting with the one or more rays; and determining a depth for the one or more cells.

8 . The computer-implemented method of claim 7 , wherein the depth is correlated to a duration of interaction with the selected portion.

9 . The computer-implemented method of claim 1 , further comprising:
generating a mask corresponding to the selected portion; and providing the mask to a content generation pipeline.

10 . The computer-implemented method of claim 1 , further comprising:
rendering the object within an interaction environment; and providing one or more tools, within the interaction environment, to identify the selected portion.

11 . The computer-implemented method of claim 10 , wherein both the scene and the object are generated objects using one or more content generation models.

12 . A processor comprising:
one or more processing units to:
generate, via one or more diffusion models, a scene representation as a neural radiance field (NeRF);
receive an input corresponding to a portion of the NeRF;
generate, based on an indication, a modified NeRF for the portion, the modified NeRF including one or more content portions different from the NeRF;
combine the modified NeRF and the NeRF to form an updated NeRF; and
provide a visual representation of the updated NeRF.

13 . The processor of claim 12 , wherein combining the modified NeRF and the NeRF corresponds to overwriting one or more feature values of the NeRF with modified feature values of the modified NeRF.

14 . The processor of claim 12 , wherein the portion of the NeRF is a three-dimensional (3D) representation generated from a two-dimensional (2D) input and one or more properties of the 2D input.

15 . The processor of claim 12 , wherein the indication is at least one of a textual prompt, an auditory prompt, or a feature associated with the input.

16 . The processor of claim 12 , wherein the processor is comprised in at least one of:
a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting virtual reality (VR) content; a system for generating or presenting augmented reality (AR) content; a system for generating or presenting mixed reality (MR) content; a system incorporating one or more Virtual Machines (VMs); a system for performing operations for a conversational Al application; a system for performing operations for a generative Al application; a system for performing operations using a language model; a system for performing one or more generative content operations using a large language model (LLM); a system implemented at least partially in a data center; a system for performing hardware testing using simulation; a system for performing one or more generative content operations using a language model; a system for synthetic data generation; a collaborative content creation platform for 3D assets; or a system implemented at least partially using cloud computing resources.

17 . A system, comprising:
one or more processors comprising processing circuitry to generate an output representation of a three-dimensional (3D) scene as a combined neural radiance field (NeRF), the combined NeRF including an original volume and a modified volume added to the original volume, the modified volume being generated for a selected volume of the original volume based, at least, on an input corresponding to content for the modified volume.

18 . The system of claim 17 , wherein objects associated with both the original volume and the modified volume are formed using one or more diffusion models.

19 . The system of claim 17 , wherein the modified volume is computed from a second input corresponding to a two-dimensional area and a property of the second input.

20 . The system of claim 17 , wherein the system comprises at least one of:
a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting virtual reality (VR) content; a system for generating or presenting augmented reality (AR) content; a system for generating or presenting mixed reality (MR) content; a system incorporating one or more Virtual Machines (VMs); a system for performing operations for a conversational Al application; a system for performing operations for a generative Al application; a system for performing operations using a language model; a system for performing one or more generative content operations using a large language model (LLM); a system implemented at least partially in a data center; a system for performing hardware testing using simulation; a system for performing one or more generative content operations using a language model; a system for synthetic data generation; a collaborative content creation platform for 3D assets; or a system implemented at least partially using cloud computing resources.

Assignees

• Nvidia Corp

Inventors

• Kreis Karsten Julian
• Shugrina Maria
• Liu Ming-Yu
• Perel Or
• Fidler Sanja
• Takikawa Towaki Alan
• Lin Tsung-Yi
• Zeng Xiaohui

Classifications

• G06T15/06 Primary

  Ray-tracing · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T15/005 Primary

  General purpose rendering architectures · CPC title