Processing speech signals of a user to generate a visual representation of the user

Patent metadata

Field	Value
Publication number	US-11900940-B2
Application number	US-202318102000-A
Country	US
Kind code	B2
Filing date	Jan 26, 2023
Priority date	Aug 13, 2018
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

A computing system for generating image data representing a speaker's face includes a detection device configured to route data representing a voice signal to one or more processors and a data processing device comprising the one or more processors configured to generate a representation of a speaker that generated the voice signal in response to receiving the voice signal. The data processing device executes a voice embedding function to generate a feature vector from the voice signal representing one or more signal features of the voice signal, maps a signal feature of the feature vector to a visual feature of the speaker by a modality transfer function specifying a relationship between the visual feature of the speaker and the signal feature of the feature vector; and generates a visual representation of at least a portion of the speaker based on the mapping, the visual representation comprising the visual feature.

First claim

What is claimed is:

1. A computing system for generating image data representing a speaker's face, the computing system comprising:
one or more processors; and
a memory storing instructions for generating a representation of a speaker based on a voice signal, the instructions, when executed by the one or more processors, configured to cause the one or more processors to perform operations comprising:
obtaining, from the voice signal, data representing one or more signal features of the voice signal;
associating a signal feature with a visual feature representing at least a portion of a physical appearance of a given speaker, the associating being based on a specified relationship between the visual feature included in the physical appearance of the given speaker and the signal feature;
reconstructing, based on the visual feature, a rendered representation of the speaker that approximates a real physical appearance of the speaker by including the visual feature;
based on the rendered representation and independent of an image representing the physical appearance of the speaker being provided to the one or more processors, generating a visual representation of at least a portion of the speaker.

2. The computing system of claim 1 , further comprising training a voice embedding function to set values for parameters of the voice embedding function that specify which of the one or more signal features of the voice signal, the training being based on one or more covariate classifiers that receive image data and voice signals.

3. The computing system of claim 1 , further comprising generating an inference of a value for the visual feature based on a known correlation of the one or more signal features of the voice signal to the visual feature of the speaker.

4. The computing system of claim 3 , where value for the visual feature comprises a size or relative proportions of articulators and vocal chambers of the speaker.

5. The computing system of claim 1 , wherein the visual representation comprises a reconstructed representation of a face of the speaker.

6. The computing system of claim 1 , wherein at least one of the one or more signal features of the voice signal comprises a voice quality feature, wherein the voice quality feature is related deterministically to measurements of a one or more vocal tract parameters for the speaker, wherein the measurements of the one or more vocal tract parameters are related to measurements of a face of the speaker, and wherein the operations comprise recreating a geometry and of the face of the speaker based on determining the voice quality feature.

7. The computing system of claim 1 , the operations further comprising receiving data comprising a template face, and modifying the data comprising the template face to incorporate the visual feature.

8. The computing system of claim 1 , where the visual feature comprises one or more of a skull structure, a gender of the speaker, an ethnicity of the speaker, a facial landmark of the speaker, or a nose structure of the speaker.

9. The computing system of claim 1 , wherein the operations further comprise:
generating a facial image of the speaker in two or three dimensions independent of receiving data comprising a template image.

10. The computing system of claim 1 , further comprising generating a statistically plausible face that incorporates the visual feature, the generating being based on a regression function.

11. The computing system of claim 1 , further comprising generating a statistically plausible face that incorporates the visual feature, the generating being based on a generative model.

12. The computing system of claim 1 , the operations further comprising receiving auxiliary data about the speaker comprising an age, a height, a gender, an ethnicity, or a body-mass index (BMI) value.

13. The computing system of claim 12 , the operations further comprising receiving one or more body indices of the speaker based on the auxiliary data, wherein the visual representation of the speaker comprises a full-body representation based on the one or more body indices.

14. The computing system of claim 13 , where the body indices are represented by a vector that includes a number of linear and volumetric characterizations of a body of the speaker.

15. The computing system of claim 13 , wherein a relation between visual features and the body indices is modelled by a neural network that is trained from training data comprising at least one of image data representing faces of speakers and voice signals.

16. A computing system for generating a voice signal, the computing system comprising:
one or more processors; and
a memory storing instructions for generating a simulation of a voice signal that approximates a real voice of a speaker, the instructions, when executed by the one or more processors, configured to cause the one or more processors to perform operations comprising:
obtaining data representing facial features of the speaker represented in an image, the facial features comprising a portion of a facial geometry;
associating the portion of the facial geometry to a signal feature of the voice signal, the associating being based on a specified relationship between the facial features of the speaker represented in the image and the signal feature of the voice signal; and
based on the associating, generating the voice signal to simulate the real voice of the speaker in the image.

17. The computing system of claim 16 , wherein the signal feature comprises at least one of a vocal resonance value, a vocal anti-resonance value, a pitch, or an estimated glottal flow waveform.

18. The computing system of claim 16 , the operations further comprising:
determining a voice quality of the voice signal comprising one or more spectral features; and
determining a style of the voice signal, a language of the voice signal, or an accent for the voice signal that includes the one or more spectral features.

19. The computing system of claim 18 , the operations further comprising mapping visual features derived from facial images to estimates of one or more subcomponents of voice quality.

20. The computing system of claim 18 , the operations further comprising determining the voice quality based on training data comprising facial image-voice quality pairs.

Assignees

• Univ Carnegie Mellon

Inventors

• Singh Rita

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G10L13/00

  Speech synthesis; Text to speech systems · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title