Modeling secondary motion based on three-dimensional models

Patent metadata

Field	Value
Publication number	US-2024169553-A1
Application number	US-202218057436-A
Country	US
Kind code	A1
Filing date	Nov 21, 2022
Priority date	Nov 21, 2022
Publication date	May 23, 2024
Grant date	—

Abstract

Techniques for modeling secondary motion based on three-dimensional models are described as implemented by a secondary motion modeling system, which is configured to receive a plurality of three-dimensional object models representing an object. Based on the three-dimensional object models, the secondary motion modeling system determines three-dimensional motion descriptors of a particular three-dimensional object model using one or more machine learning models. Based on the three-dimensional motion descriptors, the secondary motion modeling system models at least one feature subjected to secondary motion using the one or more machine learning models. The particular three-dimensional object model having the at least one feature is rendered by the secondary motion modeling system.

First claim

What is claimed is:

1 . A method comprising:
receiving, by a processing device, a plurality of three-dimensional object models representing an object; encoding, by the processing device and using one or more machine learning models, three-dimensional motion descriptors of a particular three-dimensional object model based on the plurality of three-dimensional object models; modeling, by the processing device and using the one or more machine learning models, at least one feature subjected to secondary motion based on the three-dimensional motion descriptors; and rendering, by the processing device, the particular three-dimensional object model having the at least one feature.

2 . The method of claim 1 , further comprising:
receiving, by the processing device, a plurality of digital images depicting the object; and generating, by the processing device and using an additional machine learning model, the plurality of three-dimensional object models representing the object depicted in corresponding digital images.

3 . The method of claim 2 , further comprising training, by the processing device, the additional machine learning model using training data by comparing two-dimensional representations of generated three-dimensional object models to additional two-dimensional representations of the object depicted in the corresponding digital images.

4 . The method of claim 1 , wherein the three-dimensional motion descriptors describe surface normals and velocities of corresponding portions of the particular three-dimensional object model.

5 . The method of claim 4 , wherein the surface normals of the three-dimensional motion descriptors are encoded based on spatial derivatives of the corresponding portions of the particular three-dimensional object model.

6 . The method of claim 4 , wherein the velocities of the three-dimensional motion descriptors are encoded based on temporal derivatives of the corresponding portions of the plurality of three-dimensional object models.

7 . The method of claim 1 , wherein the modeling includes generating a two-dimensional shape of the at least one feature subjected to the secondary motion based on the three-dimensional motion descriptors.

8 . The method of claim 7 , wherein the modeling includes determining surface normals of the at least one feature subjected to the secondary motion based on the two-dimensional shape and the three-dimensional motion descriptors.

9 . The method of claim 8 , wherein the modeling includes combining the two-dimensional shape of the at least one feature and the surface normals of the at least one feature.

10 . The method of claim 1 , wherein the rendering includes mapping the at least one feature subjected to the secondary motion to the particular three-dimensional object model.

11 . The method of claim 1 , wherein the plurality of three-dimensional object models are generated from a plurality of digital images depicting the object, and the one or more machine learning models are trained by comparing the particular three-dimensional object model having the at least one feature to the object depicted in a digital image from which the particular three-dimensional object model was generated.

12 . A system comprising:
a memory component; and a processing device coupled to the memory component, the processing device to perform operations including:
receiving a plurality of three-dimensional object models representing an object;
encoding, using one or more machine learning models, surface normals and velocities of corresponding portions of a particular three-dimensional object model based on the plurality of three-dimensional object models;
modeling, using the one or more machine learning models, at least one feature subjected to secondary motion based on the surface normals and the velocities; and
rendering the particular three-dimensional object model having the at least one feature.

13 . The system of claim 12 , wherein the surface normals are encoded based on spatial derivatives of the corresponding portions of the particular three-dimensional object model.

14 . The system of claim 12 , wherein the velocities are encoded based on temporal derivatives of the corresponding portions of the plurality of three-dimensional object models.

15 . The system of claim 12 , wherein the encoding includes recording the surface normals and the velocities in a two-dimensional map, each pixel in the two-dimensional map representing a corresponding portion of the particular three-dimensional object model and being encoded with a surface normal and a velocity.

16 . The system of claim 15 , wherein the encoding includes projecting the pixels of the two-dimensional map onto the particular three-dimensional object model.

17 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving three-dimensional motion descriptors relating to a particular three-dimensional object model; receiving at least one feature that is subject to secondary motion to be applied to the particular three-dimensional object model; determining surface normals of the at least one feature subjected to the secondary motion based on the three-dimensional motion descriptors; and modeling the at least one feature subjected to the secondary motion based on the surface normals.

18 . The non-transitory computer-readable medium of claim 17 , the operations further comprising generating a two-dimensional shape of the at least one feature subjected to the secondary motion based on the three-dimensional motion descriptors, the surface normals being determined based on the two-dimensional shape.

19 . The non-transitory computer-readable medium of claim 18 , wherein the modeling includes combining the two-dimensional shape of the at least one feature and the surface normals of the at least one feature.

20 . The non-transitory computer-readable medium of claim 17 , the operations further comprising mapping the at least one feature subjected to the secondary motion to the particular three-dimensional object model.

Assignees

• Adobe Inc

Inventors

• Yoon Jae Shin
• Shu Zhixin
• Wang Yangtuanfeng
• Lu Jingwan
• Yang Jimei
• Aksit Duygu Ceylan

Classifications

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title