Method, electronic device, and computer program product for generating avatar

Patent metadata

Field	Value
Publication number	US-12014454-B2
Application number	US-202217687874-A
Country	US
Kind code	B2
Filing date	Mar 7, 2022
Priority date	Jan 21, 2022
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for generating an avatar. The method includes generating an indication of correlation among image information, audio information, and text information of a video. The method may further include generating, based on the indication of the correlation, a first feature set and a second feature set representing features of a target object in the video, wherein the first feature set represents invariant features of the target object in the video, and the second feature set represents equivariant features of the target object in the video. The method may further include generating the avatar based on the first feature set and the second feature set. With this method, the generated avatar can be made more accurate and vivid with a better effect, while also reducing data annotation cost, improving operation efficiency, and enhancing user experience.

First claim

What is claimed is:

1. A method for generating an avatar, comprising:
generating an indication of correlation among image information, audio information, and text information of a video, the indication of the correlation comprising a tensor characterizing feature element associations between corresponding feature elements of the image information, the audio information, and the text information;
generating, based on the indication of the correlation and at least in part through application of a decomposition process to the tensor, a first feature set and a second feature set representing features of a target object in the video, wherein the first feature set represents invariant features of the target object in the video, and the second feature set represents equivariant features of the target object in the video; and
generating the avatar based on the first feature set and the second feature set;
wherein generating, based on the indication of the correlation, a first feature set and a second feature set representing features of a target object in the video comprises:
decomposing the tensor to obtain a decomposed image feature set, a decomposed audio feature set, and a decomposed text feature set; and
integrating the decomposed image feature set, the decomposed audio feature set, and the decomposed text feature set to generate an integrated feature set; and
wherein generating, based on the indication, a first feature set and a second feature set representing features of a target object in the video further comprises:
decomposing the integrated feature set into the first feature set and the second feature set.

2. The method according to claim 1 , wherein generating an indication of correlation among image information, audio information, and text information of a video comprises:
encoding the image information, the audio information, and the text information, respectively, to generate a corresponding image feature set, a corresponding audio feature set, and a corresponding text feature set; and
generating, based on correlation among the image feature set, the audio feature set, and the text feature set, the indication of the correlation.

3. The method according to claim 2 , wherein each element in the indication of the correlation represents correlation among elements in each of the image feature set, the audio feature set, and the text feature set at a corresponding index.

4. The method according to claim 1 , wherein the indication of the correlation comprises a synthetic tensor.

5. The method according to claim 1 , further comprising:
acquiring, based on the first feature set and the second feature set, a facial expression parameter, an attribute parameter, and a pose parameter of the target object; and
rendering a to-be-rendered object according to the acquired facial expression parameter, attribute parameter, and pose parameter to generate the avatar.

6. The method according to claim 1 , further comprising:
performing audio recognition on the audio information to obtain the text information.

7. The method according to claim 1 , wherein the image information, the audio information, and the text information are temporally consistent in the video.

8. The method according to claim 1 , further comprising:
training an avatar generation model, wherein training an avatar generation model comprises:
receiving a sample image, a sample audio, and a sample text, wherein the sample image comprises a target object;
generating an indication of correlation among the sample image, the sample audio, and the sample text;
generating, based on the indication of the correlation, a first training feature set and a second training feature set for representing features of the target object; and
training the avatar generation model based on the first training feature set and the second training feature set.

9. The method according to claim 8 , wherein generating an indication of correlation among the sample image, the sample audio, and the sample text comprises:
encoding the sample image, the sample audio, and the sample text, respectively, to generate a corresponding sample image feature set, a corresponding sample audio feature set, and a corresponding sample text feature set; and
generating, based on correlation among the sample image feature set, the sample audio feature set, and the sample text feature set, the indication of the correlation.

10. The method according to claim 9 , wherein each element in the indication of the correlation represents correlation among elements in each sample feature set of the sample image feature set, the sample audio feature set, and the sample text feature set at a corresponding index.

11. The method according to claim 8 , wherein the indication of the correlation comprises a training synthetic tensor, and wherein generating, based on the indication of the correlation, a first training feature set and a second training feature set representing features of the target object comprises:
decomposing the training synthetic tensor to generate a decomposed training image feature set, a decomposed training audio feature set, and a decomposed training text feature set; and
integrating the decomposed training image feature set, the decomposed training audio feature set, and the decomposed training text feature set to generate an integrated training feature set.

12. The method according to claim 11 , wherein generating, based on the indication of the correlation, a first training feature set and a second training feature set representing features of the target object further comprises:
decomposing the integrated training feature set into the first training feature set and the second training feature set,
wherein the first training feature set is used to represent invariant features of the target object and the second training feature set is used to represent equivariant features of the target object.

13. The method according to claim 12 , wherein training the avatar generation model based on the first training feature set and the second training feature set comprises:
performing a transform operation on the integrated training feature set to generate a transformed training feature set; and
decomposing the transformed training feature set to obtain a transformed first decomposed training feature set and a transformed second decomposed training feature set.

14. The method according to claim 13 , wherein training the avatar generation model based on the first training feature set and the second training feature set further comprises:
acquiring a first similarity loss based on the first training feature set and the transformed first decomposed training feature set;
acquiring a second similarity loss based on the second training feature set and the transformed second decomposed training feature set; and
acquiring a sum of the first similarity loss and the second similarity loss.

15. The method according to claim 14 , further comprising:
training the avatar generation model based on the sum of the first similarity loss and the second similarity loss.

16. A method for generating an avatar, comprising:
generating an indication of correlation among image information, audio information, and text information of a video;
generating, based on the indication of the correlation, a first feature set and a second feature set representing features of a target object in the video, wherein the first feature set represents invariant features of the target object in the video, and the second feature set represents equivariant features of the target object in the video; and

Assignees

• Dell Products Lp

Inventors

• Wang Zijia
• Sha Danqing
• Ni Jiacheng
• Jia Zhen

Classifications

• G06V40/174

  Facial expression recognition · CPC title

• G06V10/7747

  Organisation of the process, e.g. bagging or boosting · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title