Systems and methods for multimodal layout designs of digital publications

Patent metadata

Field	Value
Publication number	US-12536720-B2
Application number	US-202318161680-A
Country	US
Kind code	B2
Filing date	Jan 30, 2023
Priority date	Sep 16, 2022
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Embodiments described herein provide systems and methods for multimodal layout generations for digital publications. The system may receive as inputs, a background image, one or more foreground texts, and one or more foreground images. Feature representations of the background image may be generated. The foreground inputs may be input to a layout generator which has cross attention to the background image feature representations in order to generate a layout comprising of bounding box parameters for each input item. A composite layout may be generated based on the inputs and generated bounding boxes. The resulting composite layout may then be displayed on a user interface.

First claim

What is claimed is:

1 . A method of generating a visual layout for presenting content elements, the method comprising:
receiving, via a data interface, a background image and a plurality of multimodal foreground elements including at least an image and a text; generating, by an image encoder, an image representation of the image with areas covered by the plurality of multimodal foreground elements inpainted; generating, by a text encoder, a text representation of the text; generating, by a visual transformer encoder, tokenized feature representations from of the background image; generating, by attention layers of a transformer decoder that is trained by layout parameters of prior layout samples, cross attention between the image representation and a concatenation of the text representation and the feature representations; generating, by the transformer decoder, layout bounding box parameters for the foreground elements based on attention weights from the cross attention; and generating, via a user interface, the layout by overlaying the foreground elements over the background image according to the layout bounding box parameters.

2 . The method of claim 1 , further comprising:
generating variations of the layout bounding box parameters based on ensuring the foreground elements do not overlap; and generating variations of the layout by overlaying the foreground elements over the background image according to the variations of the layout bounding box parameters.

3 . The method of claim 1 ,
wherein the text includes any combination of a category label, a length, and a natural language text, and wherein the generating, by a text encoder, a text representation of the text comprises concatenating representations of the category label, the length, and the natural language text.

4 . The method of claim 1 , further comprising:
sampling a vector based on a gaussian noise distribution; encoding the sampled vector; and concatenating the encoded vector with the text representation, wherein the cross attention with the text representation is cross attention with a representation based on the concatenated encoded vector and text representation.

5 . The method of claim 4 , further comprising:
training the transformer decoder together with a layout encoder, wherein the layout encoder generates the gaussian noise distribution based on a bounding box parameter of a training layout.

6 . The method of claim 1 , further comprising:
training a conditional discriminator to predict if a layout is a layout from a training dataset or a generated layout; and training the transformer decoder to minimize an accuracy of the conditional discriminator.

7 . The method of claim 6 , further comprising:
training an auxiliary decoder to reconstruct the text based on a final feature layer of the conditional discriminator; and further training the conditional discriminator to maximize an accuracy of the auxiliary decoder.

8 . The method of claim 1 , further comprising:
training a conditional reconstructor to reconstruct the text and the image based on a final feature layer of the transformer decoder; and training the transformer decoder to maximize an accuracy of the conditional reconstructor.

9 . A system for generating a visual layout for presenting content elements, the system comprising:
a memory that stores a transformer decoder and a plurality of processor executable instructions; a communication interface that receives a background image and a plurality of multimodal foreground elements including at least an image and a text; and one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
generating, by an image encoder, an image representation of the image with areas covered by the plurality of multimodal foreground elements inpainted;
generating, by a text encoder, a text representation of the text;
generating, by a visual transformer encoder, tokenized feature representations from the background image;
generating, by attention layers of the transformer decoder that is trained by layout parameters of prior layout samples, cross attention between the image representation and a concatenation of the text representation and the feature representations;
generating, by the transformer decoder, layout bounding box parameters for the foreground elements based on attention weights from the cross attention; and
generating, via a user interface, the layout by overlaying the foreground elements over the background image according to the layout bounding box parameters.

10 . The system of claim 9 , the operations further comprising:
generating variations of the layout bounding box parameters based on ensuring the foreground elements do not overlap; and generating variations of the layout by overlaying the foreground elements over the background image according to the variations of the layout bounding box parameters.

11 . The system of claim 9 ,
wherein the text includes any combination of a category label, a length, and a natural language text, and wherein the generating, by a text encoder, a text representation of the text comprises concatenating representations of the category label, the length, and the natural language text.

12 . The system of claim 9 , the operations further comprising:
sampling a vector based on a gaussian noise distribution; encoding the sampled vector; and concatenating the encoded vector with the text representation, wherein the cross attention with the text representation is cross attention with a representation based on the concatenated encoded vector and text representation.

13 . The system of claim 12 , the operations further comprising:
training the transformer decoder together with a layout encoder, wherein the layout encoder generates the gaussian noise distribution based on a bounding box parameter of a training layout.

14 . The system of claim 9 , the operations further comprising:
training a conditional discriminator to predict if a layout is a layout from a training dataset or a generated layout; and training the transformer decoder to minimize an accuracy of the conditional discriminator.

15 . The system of claim 14 , the operations further comprising:
training an auxiliary decoder to reconstruct the text based on a final feature layer of the conditional discriminator; and further training the conditional discriminator to maximize an accuracy of the auxiliary decoder.

16 . The system of claim 9 , the operations further comprising:
training a conditional reconstructor to reconstruct the text and the image based on a final feature layer of the transformer decoder; and training the transformer decoder to maximize an accuracy of the conditional reconstructor.

17 . A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, a background image and a plurality of multimodal foreground elements including at least an image and a text; generating, by an image encoder, an image representation of the image with areas covered by the plurality of multimodal foreground elements inpainted; generating, by a text encoder, a text representation of the text; generating, by a visual transformer encoder, tokenized feature representations of from the background image; generating, by attention

Assignees

• Salesforce Inc

Inventors

• Yu Ning
• Chen Chia-Chih
• Chen Zeyuan
• Xiong Caiming
• Niebles Duque Juan Carlos
• Xu Ran
• Meng Rui

Classifications

• G06T2210/12

  Bounding box · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T9/00 Primary

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F40/126

  Character encoding · CPC title