Video synthesis via multimodal conditioning

Patent metadata

Field	Value
Publication number	US-12375766-B2
Application number	US-202217957312-A
Country	US
Kind code	B2
Filing date	Sep 30, 2022
Priority date	Feb 14, 2022
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

A multimodal video generation framework (MMVID) that benefits from text and images provided jointly or separately as input. Quantized representations of videos are utilized with a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. A new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens is used to improve video quality and consistency. Text augmentation is utilized to improve the robustness of the textual representation and diversity of generated videos. The framework incorporates various visual modalities, such as segmentation masks, drawings, and partially occluded images. In addition, the MMVID extracts visual information as suggested by a textual prompt.

First claim

What is claimed is:

1. A conditional video synthesis method, the method comprising:
accessing a multimodal video generation framework (MMVID) comprising a pretrained autoencoder, a language model, a mask-predict algorithm, and a pretrained bidirectional transformer;
receiving multimodal input signals; and
generating a video by applying the MMVID to the multimodal input signals.

2. The method of claim 1 , wherein the multimodal input signals comprise a visual control and a textual control.

3. The method of claim 2 , wherein the MMVID is a two-stage video generation framework comprising a first stage and a second stage, the method further comprising:
quantizing the visual control in the first stage using the pretrained autoencoder; and
predicting a video token in the second stage from the multimodal input signals using the pretrained bidirectional transformer.

4. The method of claim 3 , wherein the pretrained autoencoder comprises an encoder and a decoder and wherein the method further comprises obtaining a quantized representation of images using the pretrained autoencoder.

5. The method of claim 4 , wherein the pretrained bidirectional transformer is non-autoregressive.

6. The method of claim 5 , further comprising:
pretraining a bidirectional transformer on video tokens by a masked sequence estimation, a relevance estimation, and a video estimation to generate the pretrained bidirectional transformer.

7. The method of claim 1 , wherein textual control and visual control are produced by text augmentation of input text by the language model.

8. The method of claim 7 , wherein the textual control and the visual control are independent.

9. The method of claim 7 , wherein the textual control and the visual control are dependent and wherein the MMVID extracts visual information from the visual control as suggested by the textual control.

10. The method of claim 7 , wherein the visual control consists of a combination of images and videos.

11. The method of claim 7 , wherein generating the video is done by video interpolation.

12. The method of claim 7 , wherein generating the temporally video is done by video extrapolation.

13. A system, comprising;
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to perform operations comprising;
accessing a multimodal video generation framework (MMVID) comprising a pretrained autoencoder, a language model, a mask-predict algorithm, and a pretrained non-autoregressive bidirectional transformer;
receiving multimodal input signals; and
generating a video by applying the MMVID to the multimodal input signals.

14. The system of claim 13 , wherein the pretrained autoencoder comprises an encoder and a decoder, wherein the pretrained autoencoder is configured to obtain a quantized representation of images, and the pretrained non-autoregressive bidirectional transformer is pretrained on video tokens by a masked sequence estimation, a relevance estimation, and a video estimation.

15. The system of claim 13 , wherein the multimodal input signals comprise a visual control and a textual control, wherein the textual control is produced by text augmentation of input text by the language model, wherein the textual control and the visual control are independent.

16. The system of claim 13 , wherein the multimodal input signals comprise a visual control and a textual control, wherein the textual control is produced by text augmentation of input text by the language model, wherein the textual control and the visual control are dependent.

17. A non-transitory computer-readable storage medium including instruction that when executed by a processor perform operations comprising:
accessing a multimodal video generation framework (MMVID) comprising a pretrained autoencoder, a language model, a mask-predict algorithm, and a pretrained non-autoregressive bidirectional transformer;
receiving multimodal input signals; and
generating a video by applying the MMVID to the multimodal input signals.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the pretrained autoencoder comprises an encoder and a decoder, wherein the pretrained autoencoder is configured to obtain a quantized representation for images, and the pretrained non-autoregressive bidirectional transformer is pretrained on video tokens by a masked sequence estimation, a relevance estimation, and a video estimation.

19. The non-transitory computer-readable storage medium of claim 17 , wherein the multimodal input signals comprise a visual control and a textual control, wherein the textual control is produced by text augmentation of input text by the language model, wherein the textual control and the visual control are independent.

20. The non-transitory computer-readable storage medium of claim 17 , wherein the multimodal input signals comprise a visual control and a textual control, wherein the textual control is produced by text augmentation of input text by the language model, wherein the textual control and the visual control are dependent.

Assignees

• Barbieri Francesco
• Han Ligong
• Lee Hsin Ying
• Minaee Shervin
• Olszewski Kyle
• Ren Jian
• Tulyakov Sergey
• Snap Inc

Inventors

• Barbieri Francesco
• Han Ligong
• Lee Hsin-Ying
• Minaee Shervin
• Olszewski Kyle
• Ren Jian
• Tulyakov Sergey

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• H04N21/23418

  involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• H04N21/47205 Primary

  for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally · CPC title