Generating audio files from text input

Patent metadata

Field	Value
Publication number	US-2024112687-A1
Application number	US-202318477859-A
Country	US
Kind code	A1
Filing date	Sep 29, 2023
Priority date	Sep 29, 2022
Publication date	Apr 4, 2024
Grant date	—

Abstract

Methods, systems, and storage media for generating audio data includes receiving a text input. The method also includes receiving a plurality of representative audio sources and encoding the plurality of representative audio sources into a plurality of audio tokens. The method includes encoding the text input into a plurality of text representations. The method comprises mapping each audio tokens of the plurality of audio tokens to a text representation of the plurality of text representations. The method also comprises determining a relationship score based on mapping each audio tokens to the text representation, wherein the relationship score identifies a distribution of audio tokens from the plurality of audio tokens. The method and systems can also comprise decoding the subgroup of audio tokens to yield a reconstructed audio source.

First claim

What is claimed is:

1 . A computer-implemented method for generating audio data, the method comprising:
receiving a text input; receiving a plurality of representative audio sources; encoding the plurality of audio sources into a plurality of audio tokens; encoding the text input into a plurality of text representations; mapping each audio tokens of the plurality of audio tokens to a text representation of the plurality of text representations; determining a relationship score based on mapping each audio tokens to the text representation, wherein the relationship score identifies a distribution of audio tokens from the plurality of audio tokens; in response on the relationship score, determining a subgroup of audio tokens from the distribution of audio tokens; and decoding the subgroup of audio tokens to yield a reconstructed audio source.

2 . The method of claim 1 , wherein decoding the plurality of audio tokens comprises, determining a time domain loss and a frequency domain loss and implementing a Fourier transform to reduce the time domain loss and the frequency domain loss.

3 . The method of claim 1 , wherein decoding the subgroup of audio tokens to yield a reconstructed audio source comprises decompressing the subgroup of audio tokens.

4 . The method of claim 1 wherein encoding the text input is performed by a trained text encoder model.

5 . The method of claim 1 further comprising, transmitting the reconstructed audio source to a virtual reality or augment reality environment.

6 . The method of claim 1 further comprising training a compression and decompression model for the plurality of audio resources based on encoding the plurality of audio resources and decoding the subgroup of audio tokens.

7 . The method of claim 1 further comprising, transmitting the reconstructed audio source to a virtual reality or augment reality environment.

8 . A system for generating audio data, comprising:
one or more processors; a memory comprising instructions stored thereon, which when executed by the one or more processors, causes the one or more processors to perform:
receiving a text input;
receiving a plurality of representative audio sources;
encoding the plurality of representative audio sources into a plurality of audio tokens;
encoding the text input into a plurality of text representations;
mapping each audio tokens of the plurality of audio tokens to a text representation of the plurality of text representations;
determining a relationship score based on mapping each audio tokens to the text representation, wherein the relationship score identifies a distribution of audio tokens from the plurality of audio tokens;
in response to the relationship score, determining a subgroup of audio tokens from the distribution of audio tokens; and
decoding the subgroup of audio tokens to yield a reconstructed audio source, wherein decoding the subgroup of audio tokens to yield a reconstructed audio source comprises decompressing the subgroup of audio tokens.

9 . The system of claim 8 , wherein decoding the plurality of audio tokens comprises, determining a time domain loss and a frequency domain loss and implementing a Fourier transform to reduce the time domain loss and the frequency domain loss.

10 . The system of claim 8 , wherein encoding the text input is performed by a trained text encoder model.

11 . The system of claim 8 , further comprising, transmitting the reconstructed audio source to a virtual reality or augment reality environment.

12 . The system of claim 8 , further comprising training a compression and decompression model for the plurality of audio resources based on encoding the plurality of audio resources and decoding the subgroup of audio tokens.

13 . The system of claim 8 , further comprising, transmitting the reconstructed audio source to a virtual reality or augment reality environment.

14 . A non-transitory storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations for generating audio:
receiving a text input; receiving a plurality of representative audio sources; encoding the plurality of representative audio sources into a plurality of audio tokens; encoding the text input into a plurality of text representations; mapping each audio tokens of the plurality of audio tokens to a text representation of the plurality of text representations; determining a relationship score based on mapping each audio tokens to the text representation, wherein the relationship score identifies a distribution of audio tokens from the plurality of audio tokens; in response to the relationship score, determining a subgroup of audio tokens from the distribution of audio tokens; and decoding the subgroup of audio tokens to yield a reconstructed audio source.

15 . The non-transitory storage medium of claim 14 , wherein decoding the plurality of audio tokens comprises, determining a time domain loss and a frequency domain loss and implementing a Fourier transform to reduce the time domain loss and the frequency domain loss.

16 . The non-transitory storage medium of claim 14 , wherein decoding the subgroup of audio tokens to yield a reconstructed audio source comprises decompressing the subgroup of audio tokens.

17 . The non-transitory storage medium of claim 14 , wherein encoding the text input is performed by a trained text encoder model.

18 . The non-transitory storage medium of claim 14 , further comprising stored sequences of instructions, which when executed by the one or more processors, cause the one or more processors to perform, transmitting the reconstructed audio source to a virtual reality or augment reality environment.

19 . The non-transitory storage medium of claim 14 further comprising stored sequences of instructions, which when executed by the one or more processors, cause the one or more processors to perform training a compression and decompression model for the plurality of audio resources based on encoding the plurality of audio resources and decoding the subgroup of audio tokens.

20 . The non-transitory storage medium of claim 14 further comprising stored sequences of instructions, which when executed by the one or more processors, cause the one or more processors to perform, transmitting the reconstructed audio source to a virtual reality or augment reality environment.

Assignees

• Meta Platforms Tech Llc

Inventors

• Taigman Yaniv Nechemia
• Kruk Felix
• Adi Yossef Mordechay
• Synnaeve Gabriel
• Polyak Adam
• Singer Uriel
• Parikh Devi Niru
• Défossez Alexandre
• Copet Jade

Classifications

• G10L19/02

  using spectral analysis, e.g. transform vocoders or subband vocoders · CPC title

• G10L19/018 Primary

  Audio watermarking, i.e. embedding inaudible data in the audio signal · CPC title

• G10L19/0204

  using subband decomposition · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G10L13/047

  Architecture of speech synthesisers · CPC title