Low latency generative artificial intelligence audio pipeline

Patent metadata

Field	Value
Publication number	US-2025259619-A1
Application number	US-202419001278-A
Country	US
Kind code	A1
Filing date	Dec 24, 2024
Priority date	Feb 8, 2024
Publication date	Aug 14, 2025
Grant date	—

Abstract

Methods and systems are described for audio pipelines between a user and a machine learning (ML) model. The method involves obtaining digital audio data associated with an audio signal from a user's equipment. This data is then analyzed using a trained ML model, which includes one or more neural networks. The ML model determines an indication related to the digital audio data, which can be a prediction of the ending of one or more portions of the data, an intention behind the data, a key term within the data, an action to take based on the data, or the context associated with the data. Based on the determined indication, a response is generated and subsequently transmitted back to the user's equipment.

First claim

What is claimed is:

1 . A method comprising:
obtaining digital audio data associated with an audio signal from a user equipment of a user; determining, via a trained machine learning (ML) model including one or more neural networks, an indication associated with the digital audio data, wherein the indication is selected from any one or more of a prediction of an ending of one or more portions of the digital audio data, an intention of the digital audio data, a key term of the digital audio data, an action to take in view of the digital audio data, or context associated with the digital audio data; generating a response based upon the determined indication associated with the digital audio data; and transmitting the response to the user equipment of the user.

2 . The method of claim 1 , further comprising:
determining a latency associated with generating the response, wherein the latency exceeds a predetermined threshold.

3 . The method of claim 2 , further comprising:
sending, to the user based upon the latency, a masking phrase or term prior transmitting the generated response.

4 . The method of claim 1 , further comprising:
predicting one or more responses to the predicted ending of the one or more portions of the digital audio data.

5 . The method of claim 4 , further comprising:
comparing the one or more predicted responses to an entire content of the digital audio data, and selecting an optimal response from the one or more predicted responses, wherein the optimal response is the transmitted response.

6 . The method of claim 1 , wherein the predicted ending is based upon one or more of a word, context, pitch or tone associated with the user of the digital audio data.

7 . The method of claim 1 , wherein the generated response is based upon a synthesis of two or more of the determined indications.

8 . The method of claim 1 , wherein each neural network of the ML model is trained on a phoneme.

9 . The method of claim 1 , wherein the user equipment includes any one or more of a smartphone, laptop, tablet, wearable device.

10 . A system comprising:
one or more processors; and at least one memory storing instructions, that when executed by the one or more processors, cause the system to:
obtain digital audio data associated with an audio signal from a user equipment of a user;
determine, via a trained machine learning (ML) model including one or more neural networks, an indication associated with the digital audio data, wherein the indication is selected from any one or more of a prediction of an ending of one or more portions of the digital audio data, an intention of the digital audio data, a key term of the digital audio data, an action to take in view of the digital audio data, or context associated with the digital audio data;
generate a response based upon the determined indication associated with the digital audio data; and
transmit the response to the user equipment of the user.

11 . The system of claim 10 , wherein the instructions that when executed by the one or more processors, cause the system to determine a latency associated with generating the response, wherein the latency exceeds a predetermined threshold.

12 . The system of claim 11 , wherein the instructions that when executed by the one or more processors, cause the system to send, to the user based upon the latency, a masking phrase or term prior transmitting the generated response.

13 . The system of claim 10 , wherein the instructions that when executed by the one or more processors, cause the system to predict one or more responses to the predicted ending of the one or more portions of the digital audio data.

14 . The system of claim 13 , wherein the instructions that when executed by the one or more processors, cause the system to:
compare the one or more predicted responses to an entire content of the digital audio data, and select an optimal response from the one or more predicted responses, wherein the optimal response is the transmitted response.

15 . The system of claim 10 , wherein the predicted ending is based upon one or more of a word, context, pitch or tone associated with the user of the digital audio data.

16 . The system of claim 10 , wherein the generated response is based upon a synthesis of two or more of the determined indications.

17 . A method comprising:
training, via a video dataset associated with graphical stickers, a video generation machine learning (ML) model employing one or more conditioning signals; finetuning the trained video generation ML model, wherein the finetuning includes applying feedback based upon quality of motion and content to an output of the video generation ML model; and analyzing, via the trained and finetuned video generation ML model, a text string associated with a first graphical sticker input to generate a second graphical sticker.

18 . The method of claim 17 , wherein the first and second graphical stickers include any one or more of animated stickers, images, animations or videos.

19 . The method of claim 17 , wherein the video generation ML model employs image and text conditioning signals.

20 . The method of claim 19 , wherein the second graphical sticker exhibits greater motion consistency than the first graphical sticker based upon a motion conditioning signal.

Assignees

• Meta Platforms Inc

Inventors

• Khot Ishan Sanjay
• Yan David
• Basarge Shreyas Narendra
• Bagotra Ankush
• Singh Priyanshu Kumar
• Gao yue zhou
• Unterholzner John
• Huang Yuchen
• Fang Qiuren
• Venkataramani Bharath
• Yu Licheng
• Zhang Chenyun
• Bearman Amy Lawson
• Zhang Luxin
• Kalia Anmol
• Wang Dingkang
• Blanchard Elliot
• Ramchandani Ankit Rajesh
• Pumarola Peris Albert
• Liu Miao
• Schoenfeld Edgar
• Narni Krishna
• Luo Yaqiao

Classifications

• G10L2015/025

  Phonemes, fenemes or fenones being the recognition units · CPC title

• G10L13/08 Primary

  Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination · CPC title

• G10L15/063

  Training · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title