Generating consistent object views using unsupervised fine-tuning

Patent metadata

Field	Value
Publication number	US-2026024278-A1
Application number	US-202418777072-A
Country	US
Kind code	A1
Filing date	Jul 18, 2024
Priority date	Jul 18, 2024
Publication date	Jan 22, 2026
Grant date	—

Abstract

A method, apparatus, non-transitory computer readable medium, and system for image generation image generation may include obtaining a first image depicting a first view of an object, generating a second image depicting a second view of the object based on the first image, and generating a third image depicting a third view of the object based on the first image, where the third view is structurally consistent with the second view.

First claim

What is claimed is:

1 . A method comprising:
obtaining a first image depicting a first view of an object; generating, using an image generation model, a second image depicting a second view of the object based on the first image; and generating, using the image generation model, a third image depicting a third view of the object based on the first image, wherein the third view is structurally consistent with the second view.

2 . The method of claim 1 , wherein:
the image generation model is trained using unsupervised learning by generating a three-dimensional (3D) model based on an output image of the image generation model, computing a reward based on the 3D model, and updating parameters of the image generation model based on the reward.

3 . The method of claim 1 , further comprising:
combining the second image and the third image to obtain an animation of the object.

4 . The method of claim 1 , further comprising:
generating a model of the object based on the second image and the third image.

5 . The method of claim 1 , wherein obtaining the first image comprises:
obtaining a text prompt; generating the first image based on the text prompt.

6 . The method of claim 1 , wherein obtaining the first image comprises:
obtaining a preliminary image depicting the object and a background region; and masking the background region of the preliminary image to obtain the first image.

7 . A method of training a machine learning model, the method comprising:
obtaining a training set including a training image depicting a first view of an object; generating an output image depicting a second view of the object based on the training image; generating a three-dimensional (3D) model based on the output image and the training image; and training, using the 3D model, an image generation model to generate a synthetic image depicting a third view of the object.

8 . The method of claim 7 , wherein training the image generation model comprises:
generating a target image based on the 3D model; and computing a reward by comparing the target image with the output image, wherein the image generation model is trained based on the reward.

9 . The method of claim 8 , wherein:
the reward is based on a perceptual similarity metric.

10 . The method of claim 7 , wherein training the image generation model comprises:
generating a plurality of output images depicting a plurality of views of the object; and generating the 3D model based on the plurality of output images.

11 . The method of claim 10 , further comprising:
generating a target image based on the 3D model; and comparing the target image with a second output image other than the plurality of output images used to generate the 3D model.

12 . The method of claim 10 , further comprising:
computing a plurality of rewards corresponding to the plurality of output images, wherein the image generation model is trained based on the plurality of rewards.

13 . The method of claim 7 , wherein:
the 3D model comprises a Neural Radiance Field (NeRF) model.

14 . The method of claim 7 , wherein:
the training comprises reinforcement learning (RL).

15 . The method of claim 14 , wherein:
the RL comprises a Denoising Diffusion Policy Optimization (DDPO).

16 . An apparatus comprising:
at least one processor; at least one memory storing instructions executable by the at least one processor; and the apparatus further comprising an image generation model comprising parameters stored in the at least one memory, wherein the image generation model is trained to generate a synthetic image depicting a second view of an object based on an input image depicting a first view of the object, wherein the image generation model is trained using unsupervised learning by generating a three-dimensional (3D) model based on an output image of the image generation model and computing a reward based on the 3D model.

17 . The apparatus of claim 16 , further comprising:
a 3D modeling component configured to generate the 3D model.

18 . The apparatus of claim 16 , further comprising:
a rendering component configured to generate images based on the 3D model.

19 . The apparatus of claim 16 , wherein:
the image generation model comprises a diffusion model.

20 . The apparatus of claim 16 , further comprising:
a reward component configured to compute the reward.

Assignees

• Adobe Inc

Inventors

• Xie Desai
• Li Jiahao
• Tan Hao
• Sun Xin
• Shu Zhixin
• Zhou Yi
• Bi sai

Classifications

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title