Modifying poses of two-dimensional humans in two-dimensional images by reposing three-dimensional human models representing the two-dimensional humans

Patent metadata

Field	Value
Publication number	US-12394166-B2
Application number	US-202318304147-A
Country	US
Kind code	B2
Filing date	Apr 20, 2023
Priority date	Oct 6, 2022
Publication date	Aug 19, 2025
Grant date	Aug 19, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images. The disclosed systems further use three-dimensional representations of two-dimensional images to customize focal points for the two-dimensional images.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating, by at least one processor and for display within a graphical user interface of a client device, one or more interactive elements for modifying a pose of a three-dimensional human model mesh based on sampled points of a two-dimensional human image in a two-dimensional image, wherein the three-dimensional human model mesh is displayed as an overlay on the two-dimensional image;
modifying, by the at least one processor in response to an interaction with the one or more interactive elements, the pose of the three-dimensional human model mesh; and
generating, by the at least one processor, a modified two-dimensional image comprising a modified two-dimensional human image according to the modified pose of the three-dimensional human model mesh.

2. The computer-implemented method of claim 1 , wherein generating the one or more interactive elements comprises:
generating, utilizing one or more neural networks, the three-dimensional human model mesh within a three-dimensional space according to the two-dimensional human image in the two-dimensional image; and
generating the one or more interactive elements in response to generating the three-dimensional human model mesh within the three-dimensional space.

3. The computer-implemented method of claim 2 , wherein generating the three-dimensional human model mesh comprises:
determining, utilizing the one or more neural networks, a camera position corresponding to the two-dimensional image;
inserting the three-dimensional human model mesh at a location within the three-dimensional space based on the camera position corresponding to the two-dimensional image; and
providing the three-dimensional human model mesh for display within the graphical user interface of the client device with the two-dimensional image.

4. The computer-implemented method of claim 3 , wherein generating the three-dimensional human model mesh comprises:
extracting, utilizing the one or more neural networks, two-dimensional pose data from the two-dimensional human image in the two-dimensional image;
extracting, utilizing the one or more neural networks, three-dimensional pose data from the two-dimensional human image in the two-dimensional image; and
generating, at the location within the three-dimensional space, the three-dimensional human model mesh based on the two-dimensional pose data and the three-dimensional pose data.

5. The computer-implemented method of claim 2 , wherein generating the three-dimensional human model mesh comprises providing the three-dimensional human model mesh as the overlay within the graphical user interface of the client device based on a position of the two-dimensional human image in the two-dimensional image.

6. The computer-implemented method of claim 1 , wherein modifying the pose of the three-dimensional human model mesh comprises:
determining an initial pose of the three-dimensional human model mesh based on the pose of the two-dimensional human image in the two-dimensional image; and
modifying the pose of the three-dimensional human model mesh based on the initial pose of the three-dimensional human model mesh and the interaction with the one or more interactive elements.

7. The computer-implemented method of claim 6 , wherein generating the modified two-dimensional image comprises:
determining a range of motion of one or more portions of the three-dimensional human model mesh according to the initial pose of the three-dimensional human model mesh and a target pose of the three-dimensional human model mesh; and
providing, for display within the graphical user interface of the client device in connection with the interaction with the one or more interactive elements, a corresponding range of motion of one or more corresponding portions of the two-dimensional human image.

8. The computer-implemented method of claim 1 , wherein generating the modified two-dimensional image comprises:
generating the modified two-dimensional human image based on the pose of the three-dimensional human model mesh and an initial texture of the two-dimensional human image; and
generating the modified two-dimensional image comprising the modified two-dimensional human image.

9. The computer-implemented method of claim 1 , wherein generating the modified two-dimensional image comprises:
determining, within a three-dimensional space and based on the pose of the three-dimensional human model mesh, an interaction between the three-dimensional human model mesh and a three-dimensional object corresponding to a two-dimensional object in the two-dimensional image; and
generating the modified two-dimensional image representing an interaction between the modified two-dimensional human image and the two-dimensional object according to the interaction between the three-dimensional human model mesh and the three-dimensional object.

10. A system comprising:
one or more memory devices comprising a two-dimensional image; and
one or more processors configured to cause the system to:
generate, for display within a graphical user interface of a client device, a three-dimensional human model mesh based on sampled points of a two-dimensional human image of the two-dimensional image in a three-dimensional space;
generate, for display within the graphical user interface of the client device, one or more interactive elements for modifying a pose of the three-dimensional human model mesh, wherein the three-dimensional human model mesh is displayed as an overlay on the two-dimensional image;
modify, in response to an interaction with the one or more interactive elements, the pose of the three-dimensional human model mesh in the three-dimensional space; and
generate a modified two-dimensional image comprising a modified two-dimensional human image according to the modified pose of the three-dimensional human model mesh.

11. The system of claim 10 , wherein the one or more processors are configured to cause the system to:
generate, utilizing a plurality of neural networks, the three-dimensional human model mesh within the three-dimensional space according to a position of the two-dimensional human image within a scene of the two-dimensional image; and
generate the one or more interactive elements comprising one or more controls for modifying one or more portions of the three-dimensional human model mesh within the three-dimensional space.

12. The system of claim 11 , wherein the one or more processors are configured to cause the system to:
modify the pose of the three-dimensional human model mesh by modifying, in response to the interaction with the one or more interactive elements, a pose of a portion of the three-dimensional human model mesh according to the one or more controls; and
modify, within the graphical user interface of the client device, a pose of a portion of the two-dimensional human image corresponding to the portion of the three-dimensional human model mesh in connection with the interaction with the one or more interactive elements.

13. The system of claim 10 , wherein the one or more processors are configured to cause the system to modify the pose of the three-dimensional human model mesh by:
determining a motion constraint associated with a portion of the three-dimensional human model mesh based on a pose prior corresponding to the portion of the three-dimensional human model mesh; and
modifying the portion of the three-dimensional human model mesh according to the motion constraint.

14. The system of claim 10 , wherein the one or more processors are configured to cause the system to generate the modified two-

Assignees

• Adobe Inc

Inventors

• Gori Giorgio
• Zhou Yi
• Wang Yangtuanfeng
• Zhou Yang
• Singh Krishna Kumar
• Yoon Jae Shin
• Aksit Duygu Ceylan

Classifications

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title