Video-generation system with structured data-based video generation feature

Patent metadata

Field	Value
Publication number	US-12430834-B2
Application number	US-202217958397-A
Country	US
Kind code	B2
Filing date	Oct 2, 2022
Priority date	Oct 2, 2022
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

In one aspect, an example method includes (i) obtaining, by a computing system, structured data; (ii) generating, by the computing system using a natural language generator, a textual description of the structured data; (iii) transforming, by the computing system using a text-to-speech engine, the textual description of the structured data into synthesized speech; and (iv) generating, by the computing system using the synthesized speech, a synthetic video comprising the synthesized speech.

First claim

The invention claimed is:

1. A method comprising:
obtaining, by a computing system, structured data relating to an event;
generating, by the computing system using a natural language generator, a textual narrative of the event;
transforming, by the computing system using a text-to-speech engine, the textual narrative of the event into synthesized speech;
processing, by the computing system, the obtained structured data to determine that a part of the obtained structured data in the form of textual data and/or numeric data satisfies one or more conditions;
using the determination that the part of the obtained structured data in the form of textual data and/or numeric data satisfies one or more conditions, as a basis to render a graphic corresponding to the part of the obtained structured data; and
generating, by the computing system using the synthesized speech and the rendered graphic corresponding to the part of the obtained structured data,, a synthetic video comprising the synthesized speech and the generated graphic corresponding to the part of the obtained structured data,
wherein generating the synthetic video comprises:
generating a sequence of frames;
for each frame in the generated sequence of frames, outputting a first score for that given frame, wherein the first score is indicative of whether that given frame is realistic;
for the generated sequence of frames, outputting a second score, wherein the second score is indicative of whether the generated sequence of frames is realistic;
for the generated sequence of frames and the generated synthesized speech, outputting a third score indicative of whether synchronization between the generated sequence of frames and the generated synthesized speech is realistic;
determining a weighted average of the outputted first, second, and third scores;
determining that the determined weighted average exceeds a threshold; and
based on determining that the determined weighted average exceeds the threshold, using at least the generated sequence of frames and the generated synthesized speech to generate the synthetic video.

2. The method of claim 1 , further comprising obtaining a speech sample for a speaker,
wherein the text-to-speech engine transforms the textual narrative of the event into synthesized speech by the speaker using the speech sample for the speaker.

3. The method of claim 2 , wherein the synthetic video comprises one or more images and an accompanying audio track comprising the synthesized speech by the speaker.

4. The method of claim 1 , further comprising obtaining a sample video of a human speaking,
wherein generating the synthetic video comprises generating the synthetic video using the sample video of the human speaking and a video-synthesis model, and
wherein the synthetic video depicts the human speaking the synthesized speech.

5. The method of claim 4 , wherein:
the video-synthesis model is a temporal generative adversarial network having an ensemble of discriminators, and
the ensemble of discriminators are configured to perform a spatial-temporal integration of the sample video of the human speaking and the synthesized speech.

6. The method of claim 1 , wherein
the structured data comprises weather data, sports data, financial data, real estate data, or entertainment data.

7. A computing system comprising a processor and a non-transitory computer-readable medium having stored thereon program instructions that upon execution by the processor, cause performance of a set of acts comprising:
obtaining structured data relating to an event;
generating, using a natural language generator, a textual narrative of the event;
transforming, using a text-to-speech engine, the textual narrative of the event into synthesized speech;
processing the obtained structured data to determine that a part of the obtained structured data in the form of textual data and/or numeric data satisfies one or more conditions;
using the determination that the part of the obtained structured data in the form of textual data and/or numeric data satisfies one or more conditions, as a basis to render a graphic corresponding to the part of the obtained structured data; and
generating, using the synthesized speech and the rendered graphic corresponding to the part of the obtained structured data, a synthetic video comprising the synthesized speech and the generated graphic corresponding to the part of the obtained structured data,
wherein generating the synthetic video comprises:
generating a sequence of frames;
for each frame in the generated sequence of frames, outputting a first score for that given frame, wherein the first score is indicative of whether that given frame is realistic;
for the generated sequence of frames, outputting a second score, wherein the second score is indicative of whether the generated sequence of frames is realistic;
for the generated sequence of frames and the generated synthesized speech, outputting a third score indicative of whether synchronization between the generated sequence of frames and the generated synthesized speech is realistic;
determining a weighted average of the outputted first, second, and third scores;
determining that the determined weighted average exceeds a threshold; and
based on determining that the determined weighted average exceeds the threshold, using at least the generated sequence of frames and the generated synthesized speech to generate the synthetic video.

8. The computing system of claim 7 , wherein:
the set of acts further comprises obtaining a speech sample for a speaker, and
the text-to-speech engine transforms the textual narrative of the event into synthesized speech by the speaker using the speech sample for the speaker.

9. The computing system of claim 8 , wherein the synthetic video comprises one or more images and an accompanying audio track comprising the synthesized speech by the speaker.

10. The computing system of claim 7 , wherein:
the set of acts further comprises obtaining a sample video of a human speaking,
generating the synthetic video comprises generating the synthetic video using the sample video of the human speaking and a video-synthesis model, and
the synthetic video depicts the human speaking the synthesized speech.

11. The computing system of claim 10 , wherein:
the video-synthesis model is a temporal generative adversarial network having an ensemble of discriminators, and
the ensemble of discriminators are configured to perform a spatial-temporal integration of the sample video of the human speaking and the synthesized speech.

12. The computing system of claim 11 , wherein generating the synthetic video comprises determining facial expressions for the human while the human speaks the synthesized speech using a frame discriminator and a sequence discriminator.

13. The computing system of claim 11 , wherein generating the synthetic video comprises determining gestures for the human while the human speaks the synthesized speech using a frame discriminator and a sequence discriminator.

14. The computing system of claim 7 , wherein the structured data comprises weather data, sports data, financial data, real estate data, or entertainment data.

15. A non-transitory computer-readable medium having stored thereon program instructions that upon execution by a computing system, cause performance of a set of acts comprising:
obtaining structured data relating to an event;
generating, using a natural language generator, a textual narrative of the event;
transforming, using a text-to-speech engine, the textual narrative of the event into synthesize

Assignees

• Roku Inc

Inventors

• Ramesh Sunil
• Cutter Michael
• Pinkerton Charles Brian
• Levitian Karina

Classifications

• G10L13/04

  Details of speech synthesis systems, e.g. synthesiser structure or memory management · CPC title

• G06T13/80

  Two-dimensional [2D] animation, e.g. using sprites · CPC title

• G06T13/205

  driven by audio data · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title