Text-guided cameo generation

Patent metadata

Field	Value
Publication number	US-12430812-B2
Application number	US-202217950945-A
Country	US
Kind code	B2
Filing date	Sep 22, 2022
Priority date	Sep 22, 2022
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The coordinates of a face in the image are determined, and the face of the user or another person is added to the image at the location. The final image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

First claim

The invention claimed is:

1. A computer-implemented method of generating an image including an existing representation of a face of a person, for use in a conversation taking place in a messaging application, the method comprising:
receiving conversation input text from a user of a portable device that includes a display;
generating model input text from the conversation input text;
generating an image based on the model input text using a text-to-image model;
determining coordinates of a face in the generated image;
applying the existing representation of the face of the person to the generated image based on the coordinates of the face in the generated image, to generate an updated image including the existing representation of the face of the person;
displaying the updated image on the display of the portable device;
receiving user input to transmit the updated image in a message; and
transmitting, in response to receiving the user input, the updated image to a remote recipient.

2. The method of claim 1 , wherein the generating of the model input text comprises generating additional text using a creative caption function.

3. The method of claim 2 , wherein the generating of the model input text further comprises:
extracting key phrases from the additional text.

4. The method of claim 1 , further comprising:
processing the conversation input text with a safety filter to determine suitability of the conversation input text for image generation.

5. The method of claim 1 , wherein the text-to-image model has been generated from a large scale image dataset and refined by an existing collection of images for use in a conversation taking place in a messaging application.

6. The method of claim 5 , wherein text associated with images in the existing collection of images is expanded using an image-to-text module.

7. The method of claim 5 , further comprising:
animating the face of the person in the updated image.

8. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations for generating an image including an existing representation of a face of a person, for use in a conversation taking place in a messaging application, the operations comprising:
receiving conversation input text from a user of a portable device that includes a display;
generating model input text from the conversation input text;
generating an image based on the model input text using a text-to-image model;
determining coordinates of a face in the generated image;
applying the existing representation of the face of the person to the generated image based on the coordinates of the face in the generated image, to generate an updated image including the existing representation of the face of the person;
displaying the updated image on the display of the portable device;
receiving user input to transmit the updated image in a message; and
transmitting, in response to receiving the user input, the updated image to a remote recipient.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the generating of the model input text comprises generating additional text using a creative caption function.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the generating of the model input text further comprises:
extracting key phrases from the additional text.

11. The non-transitory computer-readable storage medium of claim 8 , wherein the operations further comprise:
processing the conversation input text with a safety filter to determine suitability of the conversation input text for image generation.

12. The non-transitory computer-readable storage medium of claim 8 , wherein the text-to-image model has been generated from a large scale image dataset and refined by an existing collection of images for use in a conversation taking place in a messaging application.

13. The non-transitory computer-readable storage medium of claim 12 , wherein text associated with images in the existing collection of images is expanded using an image-to-text module.

14. The non-transitory computer-readable storage medium of claim 13 , wherein text associated with the images in the existing collection of images is filtered for relevance of the text to the images in the existing collection of images, by using an image-text relevance model.

15. A computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for generating an image including an existing representation of a face of a person, for use in a conversation taking place in a messaging application, the operations comprising:
receiving conversation input text from a user of a portable device that includes a display;
generating model input text from the conversation input text;
generating an image based on the model input text using a text-to-image model;
determining coordinates of a face in the generated image;
applying the existing representation of the face of the person to the generated image based on the coordinates of the face in the generated image, to generate an updated image including the existing representation of the face of the person;
displaying the updated image on the display of the portable device;
receiving user input to transmit the updated image in a message; and
transmitting, in response to receiving the user input, the updated image to a remote recipient.

16. The computing apparatus of claim 15 , wherein the generating of the model input text comprises generating additional text using a creative caption function.

17. The computing apparatus of claim 16 , wherein the generating of the model input text further comprises:
extracting key phrases from the additional text.

18. The computing apparatus of claim 15 , wherein the text-to-image model has been generated from a large scale image dataset and refined by an existing collection of images for use in a conversation taking place in a messaging application.

19. The computing apparatus of claim 18 , wherein text associated with images in the existing collection of images is expanded using an image-to-text module.

20. The computing apparatus of claim 18 , wherein text associated with the images in the existing collection of images is filtered for relevance of the text to the images in the existing collection of images, by using an image-text relevance model.

Assignees

• Snap Inc

Inventors

• Ghosh Arnab
• Ren Jian
• Savchenkov Pavel
• Tulyakov Sergey

Classifications

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06F40/289

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F40/35

  Discourse or dialogue representation · CPC title

• H04L51/10

  Multimedia information · CPC title

• H04L51/04

  Real-time or near real-time messaging, e.g. instant messaging [IM] · CPC title