Learning a 3D scene generation model from images of a self-similar scene

Patent metadata

Field	Value
Publication number	US-12524959-B2
Application number	US-202318233458-A
Country	US
Kind code	B2
Filing date	Aug 14, 2023
Priority date	Aug 14, 2023
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

A scene modeling system accesses a set of input two-dimensional (2D) images of a three-dimensional (3D) environment, wherein the input 2D images captured from a plurality of camera orientations. The environment includes first content. The scene modeling system applies a scene generation model to the set of input 2D images to generate a 3D remix scene. Applying the scene generation model includes configuring the scene generation model using at least a 2D discriminator and a 3D discriminator. Applying the scene generation model includes transmitting, for display via a user interface, the 3D remix scene. The 3D remix scene includes second content that is different from the first content.

First claim

What is claimed is:

1 . A system comprising:
a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising:
applying a scene generation model to a set of input 2D images of first content to generate a 3D remix scene, wherein the 3D remix scene includes second content that is different from the first content, wherein applying the scene generation model includes, for each resolution of a sequence of progressively increasing resolutions:
generating a first 3D reference grid at a current resolution and a reference 2D image rendered from the first 3D reference grid;
generating a first 3D feature grid at the current resolution and a 2D image rendered from the first 3D feature grid;
configuring the scene generation model based on:
performing a first comparison, using a 2D discriminator, of an input 2D image of the set of input 2D images and the 2D reference image;
performing a second comparison, using a 3D discriminator, of the first 3D feature grid and the first 3D reference grid; and
based at least on the first comparison and the second comparison, configuring one or more parameters of the scene generation model, wherein the 3D remix scene is generated using the configured scene generation model; and

transmitting, for display via a user interface, the 3D remix scene.

2 . The system of claim 1 , wherein the set of input 2D images are captured from a plurality of camera orientations of a 3D environment, wherein the 3D environment includes first content.

3 . The system of claim 1 , the operations further comprising:
generating a second 3D reference grid at a second resolution and a second reference 2D image rendered from the second 3D reference grid, wherein the second resolution is a finer resolution than the current resolution; third comparing, to the second 3D reference grid using the 3D discriminator, a second 3D feature grid at the second resolution generated by applying the scene generation model configured with the one or more parameters to the set of input 2D images; and based on the third comparing, reconfiguring the configured one or more parameters of the scene generation model, wherein the 3D remix scene is generated by the scene generation model having the reconfigured one or more parameters.

4 . The system of claim 3 , wherein third comparing further comprises:
selecting, from the second 3D reference grid, a second 3D reference patch; selecting, from the 3D remix scene, a second 3D patch; and wherein the configured one or more parameters are reconfigured based on a comparison, using the 3D discriminator, of the second 3D patch with the second 3D reference patch.

5 . The system of claim 3 , the operations further comprising:
rendering, from the 3D remix scene, a second 2D image; selecting, from the second 2D image, a second 2D patch; rendering, from the second 3D reference grid, a second 2D reference image; selecting from the second 2D reference image, a second reference 2D patch; and reconfiguring the configured one or more parameters associated with the scene generation model based on a comparison, using the 2D discriminator, of the second 2D patch with the second reference 2D patch.

6 . The system of claim 1 , wherein the second content includes a second quantity of an object that is different from a first quantity of the object in the first content.

7 . The system of claim 1 , wherein the second content includes an object in a second position that is different from a first position of the object in the first content.

8 . The system of claim 1 , the operations further comprising:
capturing one or more 2D output images at one or more camera orientations within the 3D remix scene; and transmitting, for display via the user interface, the one or more 2D output images.

9 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
applying a scene generation model to a set of input 2D images of first content to generate a 3D remix scene, wherein the 3D remix scene includes second content that is different from the first content, wherein applying the scene generation model includes, for each resolution of a sequence of progressively increasing resolutions: generating a 3D reference grid at a current resolution and a reference 2D image rendered from the 3D reference grid; generating a 3D feature grid at the current resolution and a 2D image rendered from the 3D feature grid; configuring the scene generation model based on:
comparing, using at least a 2D discriminator and a 3D discriminator, the reference 3D grid and the 3D feature grid at the current resolution; and
based at least on the comparing, configuring one or more parameters of the scene generation model, wherein the 3D remix scene is generated using the configured scene generation model; and
transmitting, for display via a user interface, the 3D remix scene.

10 . The non-transitory computer-readable medium of claim 9 , wherein the comparing comprises:
rendering a reference 2D image from the 3D reference grid; performing a first comparison, using the 2D discriminator, an input 2D image of the set of input 2D images, and the 2D reference image; and performing a second comparison, using the 3D discriminator, of the 3D feature grid at the current resolution generated by applying the scene generation model to the set of input 2D images, to the 3D reference grid.

11 . The non-transitory computer-readable medium of claim 9 , wherein the second content includes a second quantity of an object that is different from a first quantity of the object in the first content.

12 . The non-transitory computer-readable medium of claim 9 , further comprising:
generating a second 3D reference grid at a second resolution and a second reference 2D image rendered from the second 3D reference grid, wherein the second resolution is a finer resolution than the current resolution; third comparing, to the second 3D reference grid using the 3D discriminator, a second 3D feature grid at the second resolution generated by applying the scene generation model configured with the one or more parameters to the set of input 2D images; and based on the third comparing, reconfiguring the configured one or more parameters of the scene generation model, wherein the 3D remix scene is generated by the scene generation model having the reconfigured one or more parameters.

13 . A method performed by one or more computing devices associated with a scene modeling system, comprising:
applying a scene generation model to a set of input 2D images of first content to generate a 3D remix scene, wherein the 3D remix scene includes second content that is different from the first content, wherein applying the scene generation model includes, for each resolution of a sequence of progressively increasing resolutions: generating a first 3D reference grid at a current resolution and a reference 2D image rendered from the first 3D reference grid;
generating a first 3D feature grid at the current resolution and a 2D image rendered from the first 3D feature grid;
configuring the scene generation model based on:
performing a first comparison, using a 2D discriminator, of the 2D image and the 2D reference image;
performing a second comparison, using a 3D discriminator, of the first 3D feature grid and the first 3D reference grid; and
based at least on the first comparison and the second comparison, configuring one or more parameters of the scene generation model, wherein the 3D remix s

Assignees

• Adobe Inc
• Ucl Business Ltd

Inventors

• Wang Oliver
• Karnewar Animesh
• Ritschel Tobias
• Mitra Niloy

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/0464 Primary

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T15/20

  Perspective computation · CPC title