Generating three-dimensional human models representing two-dimensional humans in two-dimensional images

Patent metadata

Field	Value
Publication number	US-2024144520-A1
Application number	US-202318304144-A
Country	US
Kind code	A1
Filing date	Apr 20, 2023
Priority date	Oct 6, 2022
Publication date	May 2, 2024
Grant date	—

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images. The disclosed systems further use three-dimensional representations of two-dimensional images to customize focal points for the two-dimensional images.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
extracting, by at least one processor utilizing one or more neural networks, two-dimensional pose data from a two-dimensional human extracted from a two-dimensional image; extracting, by the at least one processor utilizing the one or more neural networks, three-dimensional pose data and three-dimensional shape data corresponding to the two-dimensional human extracted from the two-dimensional image; and generating, by the at least one processor and within a three-dimensional space corresponding to the two-dimensional image, a three-dimensional human model representing the two-dimensional human by combining the two-dimensional pose data with the three-dimensional pose data and the three-dimensional shape data.

2 . The computer-implemented method of claim 1 , wherein extracting the two-dimensional pose data comprises extracting, utilizing a first neural network, the two-dimensional pose data comprising a two-dimensional skeleton with two-dimensional bones and annotations indicating one or more portions of the two-dimensional skeleton.

3 . The computer-implemented method of claim 2 , wherein extracting the three-dimensional pose data and the three-dimensional shape data comprises extracting, utilizing a second neural network, the three-dimensional pose data comprising a three-dimensional skeleton with three-dimensional bones and the three-dimensional shape data comprising a three-dimensional mesh according to the two-dimensional human.

4 . The computer-implemented method of claim 3 , wherein extracting the three-dimensional pose data comprises extracting, utilizing a third neural network for hand-specific bounding boxes, three-dimensional hand pose data corresponding to one or more hands of the two-dimensional human.

5 . The computer-implemented method of claim 1 , wherein generating the three-dimensional human model comprises iteratively adjusting one or more bones in the three-dimensional pose data according to one or more corresponding bones in the two-dimensional pose data.

6 . The computer-implemented method of claim 1 , wherein generating the three-dimensional human model comprises iteratively connecting one or more hand skeletons with three-dimensional hand pose data to a body skeleton with three-dimensional body pose data.

7 . The computer-implemented method of claim 1 , further comprising:
generating, in response to an indication to modify a pose of the two-dimensional human within the two-dimensional image, a modified three-dimensional human model with modified three-dimensional pose data; and generating a modified two-dimensional image comprising a modified two-dimensional human based on the modified three-dimensional human model.

8 . The computer-implemented method of claim 7 , further comprising:
determining, in response to the three-dimensional human model comprising the modified three-dimensional pose data, an interaction between the modified three-dimensional human model and an additional three-dimensional model within the three-dimensional space corresponding to the two-dimensional image; and generating the modified two-dimensional image comprising the modified two-dimensional human according to the interaction between the modified three-dimensional human model and the additional three-dimensional model.

9 . The computer-implemented method of claim 1 , wherein extracting the two-dimensional pose data comprises:
generating a cropped image corresponding to a boundary of the two-dimensional human in the two-dimensional image; and extracting the two-dimensional pose data from the cropped image utilizing the one or more neural networks.

10 . A system comprising:
one or more memory devices comprising a two-dimensional image; and one or more processors configured to cause the system to: extract, utilizing one or more neural networks, two-dimensional pose data corresponding to a two-dimensional skeleton for a two-dimensional human extracted from the two-dimensional image; extract, utilizing the one or more neural networks, three-dimensional pose data and three-dimensional shape data corresponding to a three-dimensional skeleton for the two-dimensional human extracted from the two-dimensional image; and generate, within a three-dimensional space corresponding to the two-dimensional image, a three-dimensional human model representing the two-dimensional human by refining the three-dimensional skeleton of the three-dimensional pose data according to the two-dimensional skeleton of the two-dimensional pose data and the three-dimensional shape data.

11 . The system of claim 10 , wherein the one or more processors are configured to cause the system to:
extract the two-dimensional pose data from the two-dimensional image utilizing a first neural network of the one or more neural networks; and extract the three-dimensional pose data and the three-dimensional shape data utilizing a second neural network of the one or more neural networks.

12 . The system of claim 10 , wherein the one or more processors are configured to cause the system to extract the three-dimensional pose data by:
generating a body bounding box corresponding to a body portion of the two-dimensional human; and extracting, utilizing a neural network, three-dimensional pose data corresponding to the body portion of the two-dimensional human according to the body bounding box.

13 . The system of claim 12 , wherein the one or more processors are configured to cause the system to extract the three-dimensional pose data by:
generating one or more hand bounding boxes corresponding to one or more hands of the two-dimensional human; and extracting, utilizing an additional neural network, additional three-dimensional pose data corresponding to the one or more hands of the two-dimensional human according to the one or more hand bounding boxes.

14 . The system of claim 13 , wherein the one or more processors are configured to cause the system to generate the three-dimensional human model by combining the three-dimensional pose data corresponding to the body portion of the two-dimensional human with the additional three-dimensional pose data corresponding to the one or more hands of the two-dimensional human.

15 . The system of claim 10 , wherein the one or more processors are configured to cause the system to generate the three-dimensional human model by iteratively modifying positions of bones in the three-dimensional skeleton based on positions of bones in the two-dimensional skeleton.

16 . The system of claim 10 , wherein the one or more processors are configured to cause the system to generate a modified two-dimensional image by:
modifying a pose of the three-dimensional human model within the three-dimensional space; generating a modified pose of the two-dimensional human within the two-dimensional image according to the pose of the three-dimensional human model in the three-dimensional space; and generating, utilizing the one or more neural networks, the modified two-dimensional image comprising a modified two-dimensional human according to the modified pose of the two-dimensional human and a camera position associated with the two-dimensional image.

17 . A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
extracting, utilizing one or more neural networks, two-dimensional pose data from a two-dimensional human extracted from a two-dimensional image; extracting, utili

Assignees

• Adobe Inc

Inventors

• Gori Giorgio
• Zhou Yi
• Wang Yangtuanfeng
• Zhou Yang
• Singh Krishna Kumar
• Yoon Jae Shin
• Aksit Duygu Ceylan

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30196

  Human being; Person · CPC title