Text to video generation

Patent metadata

Field	Value
Publication number	US-2024155071-A1
Application number	US-202318477887-A
Country	US
Kind code	A1
Filing date	Sep 29, 2023
Priority date	Sep 29, 2022
Publication date	May 9, 2024
Grant date	—

Abstract

A method and system for text-to-video generation. The method includes receiving a text input, generating a representation frame based on the text input using a model trained on text-image pairs, generating a set of frames based on the representation frame and a first frame rate, interpolating the set of frames to a higher frame rate, generating a first video based on the interpolated set of frames, increasing a resolution of the first video based on a first and second super-resolution model, and generating an output video based on a result of the super-resolution models.

First claim

What is claimed is:

1 . A computer-implemented method, performed by at least one processor, for text-to-video generation, the method comprising:
receiving a text input; generating a representation frame based on text embeddings of the text input; generating a set of frames based on the representation frame and a first frame rate; interpolating the set of frames based on a second frame rate; generating a first video based on the interpolated set of frames; increasing a resolution of the first video based on spatiotemporal information to generate a second video; and generating an output video by increasing a resolution of the second video based on spatial information.

2 . The computer-implemented method of claim 1 , wherein the set of frames are combined to generate a low-resolution video.

3 . The computer-implemented method of claim 1 , wherein the second frame rate is greater than the first frame rate.

4 . The computer-implemented method of claim 1 , wherein the representation frame is generated using a text-to-image model trained on text-image pair datasets.

5 . The computer-implemented method of claim 1 , wherein the resolution of the second video is increased on a frame-by-frame basis.

6 . The computer-implemented method of claim 1 , wherein interpolating the set of frames further comprises:
identifying specific frames from the set of frames based on at least one of a preset number of frames and a position of the frames; and generating additional frames based on the specific frames using an interpolation model to interpolate the specific frames.

7 . The computer-implemented method of claim 1 , wherein the second video is generated based on a first super-resolution model and the output video is generated based on a second super-resolution model, the first super-resolution model is fine-tuned on unlabeled video data, and the second super-resolution model applies a fixed noise to each frame in the second video.

8 . The computer-implemented method of claim 7 , wherein the first super-resolution model includes spatiotemporal convolution and attention layers, wherein a temporal convolution layer is stacked on each spatial convolution layer, and a temporal attention layer is stacked on each spatial attention layer.

9 . The computer-implemented method of claim 8 , further comprising:
initializing a temporal projection of the temporal attention layer to zero; and initializing the temporal convolution layer as an identify function.

10 . A system for text-to-video generation, the system comprising:
one or more processors; and a memory storing instructions which, when executed by the one or more processors, cause the system to:
receive a text input;
generate a representation frame based on text embeddings of the text input;
generate a set of frames based on the representation frame and a first frame rate;
interpolate the set of frames based on a second frame rate;
generate a first video based on the interpolated set of frames;
increase a resolution of the first video based on spatiotemporal information using a first super-resolution model to generate a second video; and
generate an output video by increasing a resolution of the second video based on spatial information using a second super-resolution model.

11 . The system of claim 10 , further comprising generating a low-resolution video based on the set of frames.

12 . The system of claim 10 , wherein the second frame rate is greater than the first frame rate.

13 . The system of claim 10 , wherein the representation frame is generated using a text-to-image model trained on text-image pair datasets.

14 . The system of claim 10 , wherein the first super-resolution model is fine-tuned over unlabeled video data.

15 . The system of claim 10 , wherein the resolution of the second video is increased on a frame-by-frame basis.

16 . The system of claim 10 , wherein the one or more processors further execute instructions to:
identify specific frames from the set of frames based on at least one of a preset number of frames and a position of the frames; and generate additional frames based on the specific frames using an interpolation model to interpolate the specific frames.

17 . The system of claim 10 , wherein the second super-resolution model applies a fixed noise to each frame in the second video.

18 . The system of claim 10 , wherein the first super-resolution model includes spatiotemporal convolution and attention layers, wherein a temporal convolution layer is stacked on each spatial convolution layer, and a temporal attention layer is stacked on each spatial attention layer.

19 . The system of claim 18 , wherein the one or more processors further execute instructions to:
initialize a temporal projection of the temporal attention layer to zero; and initialize the temporal convolution layer as an identity function.

20 . A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for text-to-video generation and cause the one or more processors to:
receive a text input; generate a representation frame based on text embeddings of the text input; decoding a set of frames conditioned on the representation frame and a frame rate; interpolate the set of frames based on a second frame rate; generate a first video based on the interpolated set of frames; increase a resolution of the first video based on spatiotemporal information using a first super-resolution model to generate a second video; and increase a resolution of the second video based on spatial information using a second super-resolution model to generate an output video.

Assignees

• Meta Platforms Tech Llc

Inventors

• Gupta Sonal
• Polyak Adam
• Hayes Thomas Falstad
• Yin Xi
• An Jie
• Yang Chao
• Ashual Oron
• Gafni Oran
• Parikh Devi Niru
• Taigman Yaniv Nechemia
• Singer Uriel
• Zhang Songyang
• Hu Qiyuan

Classifications

• H04N7/0117 Primary

  involving conversion of the spatial resolution of the incoming video signal (for graphics images G09G2340/0407) · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• H04N7/013

  the incoming video signal comprising different parts having originally different frame rate, e.g. video and graphics · CPC title

• H04N7/0135

  involving interpolation processes (interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59) · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title