Method and device for synthesizing motion sequence of virtual object

Patent metadata

Field	Value
Publication number	US-11978142-B2
Application number	US-202217680706-A
Country	US
Kind code	B2
Filing date	Feb 25, 2022
Priority date	Mar 4, 2020
Publication date	May 7, 2024
Grant date	May 7, 2024

Abstract

A method, device, and apparatus for synthesizing a motion sequence of a virtual object and a non-transitory computer-readable storage medium are disclosed. The method can include obtaining description information of a motion sequence of a virtual object, and determining a continuous motion clip set similar to at least some motions in the motion sequence based on the description information and a continuous motion clip library constructed with video materials. The method can further include synthesizing the motion sequence of the virtual object based on the continuous motion clip set, where each continuous motion clip in the continuous motion clip library includes a unique identifier of the continuous motion clip, motion information of the continuous motion clip, and a representation vector corresponding to the each continuous motion clip.

First claim

What is claimed is:

1. A method for synthesizing a motion sequence of a virtual object, the method comprising:
obtaining, by processing circuitry, description information of a motion sequence of the virtual object;
determining, by processing circuitry, a continuous motion clip set corresponding to at least some motions in the motion sequence based on the description information and a continuous motion clip library constructed with video materials; and
synthesizing, by processing circuitry, the motion sequence of the virtual object based on motion information of an end frame of a first continuous motion clip and motion information of a start frame of a second continuous motion clip of adjacent continuous motion clips in the continuous motion clip set, wherein
each continuous motion clip in the continuous motion clip library includes a unique identifier of the respective continuous motion clip, the motion information extracted from the respective continuous motion clip, and a representation vector corresponding to the respective continuous motion clip, and
the synthesizing includes concatenating the first continuous motion clip and the second continuous motion clip.

2. The method for synthesizing the motion sequence of the virtual object according to claim 1 , wherein the determining the continuous motion clip set further comprises:
obtaining a representation vector of the motion sequence of the virtual object based on the description information, the representation vector of the motion sequence of the virtual object linking semantic information and motion attribute information of the motion sequence; and
determining the continuous motion clip set in the continuous motion clip library that corresponds to the at least some motions in the motion sequence based on the representation vector of the motion sequence of the virtual object.

3. The method for synthesizing the motion sequence of the virtual object according to claim 2 , wherein the determining the continuous motion clip set in the continuous motion clip library corresponding to the at least some motions in the motion sequence further comprises:
obtaining a representation vector of a first motion sequence clip based on the representation vector of the motion sequence of the virtual object;
obtaining the first continuous motion clip based on the representation vector of the first motion sequence clip, a representation vector of the first continuous motion clip having a first similarity with the representation vector of the first motion sequence clip,
wherein the first similarity is greater than or equal to a similarity between a representation vector of another continuous motion clip in the continuous motion clip library and the representation vector of the first motion sequence clip.

4. The method for synthesizing the motion sequence of the virtual object according to claim 3 , wherein the determining the continuous motion clip set in the continuous motion clip library corresponding to the at least some motions in the motion sequence further comprises:
obtaining a representation vector of a second motion sequence clip based on the representation vector of the motion sequence of the virtual object, the second motion sequence clip being a next motion sequence clip of the first motion sequence clip; and
obtaining the second continuous motion clip based on motion information of a plurality of end frames of the first continuous motion clip and the representation vector of the second motion sequence clip, a representation vector of the second continuous motion clip having a second similarity with the representation vector of the second motion sequence clip,
wherein the second similarity is greater than or equal to a similarity between a representation vector of another continuous motion clip in the continuous motion clip library and the representation vector of the second motion sequence clip.

5. The method for synthesizing the motion sequence of the virtual object according to claim 1 , further comprising retargeting the synthesized motion sequence to the virtual object.

6. The method for synthesizing the motion sequence of a virtual object according to claim 1 , wherein construction of the continuous motion clip library further comprises:
extracting the motion information, motion attribute information, and semantic information of at least one moving object from the video materials;
dividing the video materials into a plurality of continuous motion clips based on the motion information of the at least one moving object;
converting motion attribute information and semantic information corresponding to each continuous motion clip into the representation vector corresponding to the respective continuous motion clip; and
storing the representation vector corresponding to the each continuous motion clip and the motion information correspondingly.

7. The method for synthesizing the motion sequence of the virtual object according to claim 6 , wherein:
the motion information includes at least one of a position, a speed, or an acceleration of each motion portion of the at least one moving object; and
the motion attribute information includes at least one of a motion rhythm or a motion mood of the at least one moving object,
the motion attribute information and the semantic information being extracted from audio data and text data of the video materials.

8. The method for synthesizing the motion sequence of the virtual object according to claim 6 , further comprising performing dimension reduction on the extracted motion information of the moving object.

9. The method for synthesizing the motion sequence of the virtual object according to claim 6 , wherein the dividing the video materials further comprises:
determining a motion state of the at least one moving object in each frame of the video materials; and
dividing the video materials into the plurality of continuous motion clips by using at least one specific frame as a dividing point, when a motion state of the at least one moving object in the at least one specific frame is stationary.

10. The method for synthesizing the motion sequence of the virtual object according to claim 1 , wherein the synthesizing the motion sequence of the virtual object further comprises:
concatenating the continuous motion clip set using timing information of the motion sequence of the virtual object,
wherein the concatenating includes fusing motion information of a plurality of end frames of a previous continuous motion clip and motion information of a plurality of starting frames of a current continuous motion clip in adjacent continuous motion clips.

11. An apparatus for synthesizing a motion sequence of a virtual object, comprising:
a processor; and
a memory that stores non-transitory computer instructions that, when executed by the processor, cause the apparatus to implement the method according to claim 1 .

12. A non-transitory computer-readable storage medium that stores computer instructions that, when executed by a processor, cause an electronic device including the processor to implement the method according to claim 1 .

13. The non-transitory computer-readable storage medium according to claim 12 , wherein the determining the continuous motion clip set corresponding to at least some motions in the motion sequence further comprises:
obtaining a representation vector of the motion sequence of the virtual object based on the description information, the representation vector of the motion sequence of the virtual object linking semantic information and motion attribute information of the motion sequence; and
determining the continuous motion clip

Assignees

• Tencent Tech Shenzhen Co Ltd
• Tencent America LLC

Inventors

• Kang Di
• Li Jing
• Chen Weidong
• Zhang Wei
• Qiao Liang
• Bao Linchao

Classifications

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T7/251

  involving models · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title