Machine learning in augmented reality content items

Patent metadata

Field	Value
Publication number	US-12165244-B2
Application number	US-202217974400-A
Country	US
Kind code	B2
Filing date	Oct 26, 2022
Priority date	Jun 10, 2020
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

Systems and methods herein describe receiving an image via an image capture device, using a machine learning model, generating an image augmentation decision, accessing an augmented reality content item, associating the generated image augmentation decision with the augmented reality content item, modifying the received image using the augmented reality content item and the associated image augmentation decision, and causing presentation of the modified image on a graphical user interface of a computing device.

First claim

What is claimed is:

1. A method comprising:
receiving, using one or more processors, an image via an image capture device;
using a first machine learning model comprising a first model type, generating an image augmentation decision, the image augmentation decision based on the received image and non-image data received from one or more sensors of the image capture device;
using a second machine learning model comprising a second model type, generating a supplemental image augmentation decision, the supplemental augmentation decision based on the image augmentation decision, the second model type different than the first model type;
accessing an augmented reality experience based on the image augmentation decision and the supplemental augmentation decision; and
modifying the image using the augmented reality experience, the image augmentation decision, and the supplemental augmentation decision.

2. The method of claim 1 , wherein the non-image data comprises one or more of location data and audio data.

3. The method of claim 1 , further comprising:
providing the generated image augmentation decision as an input to the augmented reality experience.

4. The method of claim 1 , wherein one or both of the first machine learning model and the second machine learning model is accessed from a resource library.

5. The method of claim 1 , wherein the first model type and the second model type comprise one or more of:
a classification model that is configured to provide a probability of an input to be within a predefined category;
a segmentation model that is configured to filter a portion of the image based on predefined criteria; and
a saliency model that is configured to predict points of interests within the image.

6. The method of claim 1 , wherein the first model type is a segmentation model that is configured to filter a portion of the image based on predefined criteria and the second model type is one of a classification model that is configured to provide a probability of an input to be within a predefined category and a saliency model that is configured to predict points of interests within the image.

7. The method of claim 1 , wherein the first model type is a segmentation model that is configured to filter a portion of the image based on predefined criteria and the second machine learning model is configured to determine one or both of pattern and texture data of the portion filtered by the first machine learning model.

8. The method of claim 1 , wherein the augmented reality experience is automatically accessed without user intervention.

9. The method of claim 1 , wherein the augmented reality experience is an augmented reality content item that is configured to modify image content of the received image.

10. A system, the system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, using one or more processors, an image via an image capture device;
using a first machine learning model comprising a first model type, generating an image augmentation decision, the image augmentation decision based on the received image and non-image data received from one or more sensors of the image capture device;
using a second machine learning model comprising a second model type, generating a supplemental image augmentation decision, the supplemental augmentation decision based on the image augmentation decision, the second model type different than the first model type;
accessing an augmented reality experience based on the image augmentation decision and the supplemental augmentation decision; and
modifying the image using the augmented reality experience, the image augmentation decision, and the supplemental augmentation decision.

11. The system of claim 10 , further comprising:
providing the generated image augmentation decision as an input to the augmented reality experience.

12. The system of claim 10 , wherein one or both of the first machine learning model and the second machine learning model is accessed from a resource library.

13. The system of claim 10 , wherein the first model type and the second model type comprise one or more of:
a classification model that is configured to provide a probability of an input to be within a predefined category;
a segmentation model that is configured to filter a portion of the image based on predefined criteria; and
a saliency model that is configured to predict points of interest within the image.

14. The system of claim 10 , wherein the first model type is a segmentation model that is configured to filter a portion of the image based on predefined criteria and the second model type is one of a classification model that is configured to provide a probability of an input to be within a predefined category and a saliency model that is configured to predict points of interests within the image.

15. The system of claim 10 , wherein the first model type is a segmentation model that is configured to filter a portion of the image based on predefined criteria and the second machine learning model is configured to determine one or both of pattern and texture data of the portion filtered by the first machine learning model.

16. A non-transitory computer-readable storage medium storing instructions that when executed by one or more processors of a machine, cause the computer-readable storage medium to perform operations comprising:
receiving, using one or more processors, an image via an image capture device;
using a first machine learning model comprising a first model type, generating an image augmentation decision, the image augmentation decision based on the received image and non-image data received from one or more sensors of the image capture device;
using a second machine learning model comprising a second model type, generating a supplemental image augmentation decision, the supplemental augmentation decision based on the image augmentation decision, the second model type different than the first model type;
accessing an augmented reality experience based on the image augmentation decision and the supplemental augmentation decision; and
modifying the image using the augmented reality experience, the image augmentation decision, and the supplemental augmentation decision.

17. The computer-readable storage medium of claim 16 , further comprising:
providing the generated image augmentation decision as an input to the augmented reality experience.

18. The computer-readable storage medium of claim 16 , wherein one or both of the first machine learning model and the second machine learning model is accessed from a resource library.

19. The computer-readable storage medium of claim 16 , wherein the first model type and the second model type comprise one or more of:
a classification model that is configured to provide a probability of an input to be within a predefined category;
a segmentation model that is configured to filter a portion of the image based on predefined criteria; and
a saliency model that is configured to predict points of interests within the image.

20. The computer-readable storage medium of claim 16 , wherein the first model type is a segmentation model that is configured to filter a portion of the image based on predefined criteria and the second model type is one of a classification model that is configured to provide a probability of an input to be within a predefined category and a salien

Assignees

• Snap Inc

Inventors

• Rykhliuk Olha
• Solichin Jonathan
• Stoliar Aleksei

Classifications

• G06F18/24

  Classification techniques · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T2210/16

  Cloth · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title