Method and apparatus for virtual avatar generation

Patent metadata

Field	Value
Publication number	US-12536739-B2
Application number	US-202318450428-A
Country	US
Kind code	B2
Filing date	Aug 16, 2023
Priority date	Aug 16, 2023
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

An apparatus for virtual avatar generation is provided, including a face encoder, a face fusion engine, a voice encoder, and a voice fusion engine. The face encoder extracts first facial features from the first facial image and then encodes they into M first facial feature vectors and extracts second facial features from the second facial image and then encodes they into M second facial feature vectors. The face fusion engine generates a facial fused feature vector from the facial feature vectors, thereby generating a fused image. The voice encoder extracts first voice features from the first audio clip and then encodes they into N first voice feature vectors and extracts second voice features from the second audio clip and then encodes they into N second voice feature vectors. The voice fusion engine synthesizes a voice fused feature vector from the voice feature vectors, thereby generating a fused voice.

First claim

What is claimed is:

1 . An apparatus for virtual avatar generation, comprising:
a face recorder configured to receive and store a first facial image and a second facial image; a face encoder configured to:
identify faces of the first facial image and the second facial image;
extract a plurality of first facial features from the first facial image and then encode them into M first facial feature vectors; and
extract a plurality of second facial features from the second facial image and then encode them into M second facial feature vectors;
a face fusion engine configured to synthesize a facial fused feature vector from the first facial feature vectors in combination with the second facial feature vectors, thereby generating a fused image from the facial fused feature vector; a voice recorder configured to receive and store a first audio clip and a second audio clip; a voice encoder configured to:
identify voices of the first audio clip and the second audio clip;
extract a plurality of first voice features from the first audio clip and then encode them into N first voice feature vectors; and
extract a plurality of second voice features from the second audio clip and then encode them into N second voice feature vectors;
a voice fusion engine configured to synthesize a voice fused feature vector from the first voice feature vectors in combination with the second voice feature vectors, thereby generating a fused voice from the voice fused feature vector, wherein the facial fused feature vector is generated from the first facial feature vectors and the second facial feature vectors according to a first weight distribution, and the voice fused feature vector is generated from the first voice feature vectors and the second voice feature vectors according to a second weight distribution; wherein the facial fused feature vector is generated from the first facial feature vectors and the second facial feature vectors according to a first weight distribution, and the voice fused feature vector is generated from the first voice feature vectors and the second voice feature vectors according to a second weight distribution; a database storing a plurality of face image references and audio clip references; and an automatic weight calculator configured to calculate the first weight distribution and the second weight distribution by matching the fused image with the fused voice based on pairs of the face image references and audio clip references, wherein the automatic weight calculator is further configured to:
determine a pair of intermediate weights for a set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors by using a pair of the face image references and the audio clip references;
calculate a loss function using the set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors, the intermediate weights, and the pair of the face image references and the audio clip references;
calculate the loss function for all sets of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors, the face image references, and the audio clip references, aiming to identify the minimum loss function among them; and
output determined weights, which result in the minimum loss function, for determining the first weight distribution and the second weight distribution to the face fusion engine and the voice fusion engine.

2 . The apparatus of claim 1 , further comprising a user interface that has a virtual dashboard with:
M controllable bars representing facial weights to be assigned to the first facial feature vectors and the second facial feature vectors, wherein the first weight distribution is produced by a series of the facial weights; and N controllable bars representing voice weights to be assigned to the first voice feature vectors and the second voice feature vectors, wherein the second weight distribution is produced by a series of the voice weights.

3 . The apparatus of claim 2 , wherein the M first facial feature vectors are paired one-to-one with the M second facial feature vectors, and the N first voice feature vectors are paired one-to-one with the N second voice feature vectors.

4 . The apparatus of claim 3 , wherein a first vector of the first facial feature vectors is paired with a first vector of the second facial feature vectors by a first facial weight set, wherein a second vector of the first facial feature vectors is paired with a second vector of the second facial feature vectors by a second facial weight set permitted to be different than the first facial weight set.

5 . The apparatus of claim 1 , further comprising a user interface that has an electronic display configured to dynamically show the fused image and play the fused voice.

6 . The apparatus of claim 1 , further comprising an avatar generator configured to generate a virtual avatar incorporating the fused image and the fused voice.

7 . The apparatus of claim 6 , further comprising a cache configured to store the fused image and the fused voice to be accessed by the avatar generator.

8 . A method for virtual avatar generation, comprising:
receiving and storing a first facial image and a second facial image by a face recorder; identifying faces of the first facial image and the second facial image by a face encoder; extracting a plurality of first facial features from the first facial image and then encode them into M first facial feature vectors by the face encoder; extracting a plurality of second facial features from the second facial image and then encode them into M second facial feature vectors by the face encoder; synthesizing a facial fused feature vector from the first facial feature vectors in combination with the second facial feature vectors by a face fusion engine, thereby generating a fused image from the facial fused feature vector; receiving and storing a first audio clip and a second audio clip by a voice recorder; identifying voices of the first audio clip and the second audio clip by a voice encoder; extracting a plurality of first voice features from the first audio clip and then encode them into N first voice feature vectors by the voice encoder; extracting a plurality of second voice features from the second audio clip and then encode them into N second voice feature vectors by the voice encoder; synthesizing a voice fused feature vector from the first voice feature vectors in combination with the second voice feature vectors by a voice fusion engine, thereby generating a fused voice from the voice fused feature vector, wherein the facial fused feature vector is generated from the first facial feature vectors and the second facial feature vectors according to a first weight distribution, and the voice fused feature vector is generated from the first voice feature vectors and the second voice feature vectors according to a second weight distribution; storing a plurality of face image references and audio clip references in a database; calculating, by an automatic weight calculator, the first weight distribution and the second weight distribution by matching the fused image with the fused voice based on pairs of the face image references and audio clip references; determining, by the automatic weight calculator, a pair of intermediate weights for a set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors by using a pair of the face image references and the audio clip references; calculating, by the automatic weight ca

Assignees

• Hong Kong Applied Science & Tech Research Inst Co Ltd

Inventors

• Li Zhuoxuan
• Xu Jingyi
• Li Yi
• Wong Hon Wah
• Wang Yanchen
• Chan Man Choi

Classifications

• G06T2207/30201

  Face · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title