Method, system, and medium for augmented reality product recommendations

Patent metadata

Field	Value
Publication number	US-12412205-B2
Application number	US-202117565648-A
Country	US
Kind code	B2
Filing date	Dec 30, 2021
Priority date	Dec 30, 2021
Publication date	Sep 9, 2025
Grant date	Sep 9, 2025

Abstract

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a video that includes a depiction of a real-world object in a real-world environment; determining a classification for the real-world environment by processing the real-world object depicted in the video; selecting an augmented reality (AR) item based on the classification of the real-world environment and the real-world object depicted in the video; modifying pixels corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object; and adding the AR item to the modified video at a display position corresponding to the modified pixels.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, a video that includes a depiction of a real-world object in a real-world environment;
determining, by a trained neural network, a classification for the real-world environment by processing the real-world object depicted in the video;
selecting an augmented reality (AR) item based on the classification of the real-world environment and based on the real-world object depicted in the video;
modifying pixel data corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object;
adding a depiction of the AR item to the modified video at a display position corresponding to the modified pixel data, the adding the depiction of the AR item to the modified video comprising:
calculating characteristic points for a set of elements of the real-world object to generate a mesh based on the calculated characteristic points;
generating one or more areas on the mesh of the real-world object;
aligning a position of the one or more areas of the real-world object with one or more elements of the AR item; and
modifying one or more visual properties of the one or more areas to cause a user device to display the AR item within the video at an individual display position relative to the display position of the real-world object that has had pixel data modified;
detecting input that moves the AR item to a new position in the video;
determining that the AR item in the new position no longer overlaps a portion of the real-world object that has had the corresponding pixel data modified; and
in response to determining that the AR item in the new position no longer overlaps the portion of the real-world object that has had the corresponding pixel data modified, undoing modification of the pixel data to return pixel values of the real-world object to an original value in the video.

2. The method of claim 1 , further comprising generating, for display, the modified video with the depiction of the AR item that has been added.

3. The method of claim 1 , further comprising blurring a region corresponding to the modified pixel data, wherein the depiction of the AR item is added to the blurred region.

4. The method of claim 3 , further comprising blending pixel values in the blurred region with pixel values of other real-world objects that are adjacent to the blurred region.

5. The method of claim 1 , further comprising applying a machine learning technique to the video to modify the pixel data and generate the modified video.

6. The method of claim 5 , wherein the machine learning technique is trained to establish a relationship between different types of real-world objects and image blending patterns.

7. The method of claim 6 , further comprising training the machine learning technique by:
receiving training data comprising a plurality of training images and ground truth room blending patterns for each of the plurality of training images, each of the plurality of training images depicting a different real-world environment having different real-world object types;
selecting a first real-world object depicted in a first training image of the plurality of training images;
applying the neural network to the first training image and the first real-world object to estimate a blending pattern for the real-world environment depicted in the first training image;
computing a deviation between the estimated blending pattern and the ground truth room blending pattern associated with the first training image;
updating parameters of the neural network based on the computed deviation; and
repeating the applying, computing and updating steps for a set of the plurality of training images.

8. The method of claim 1 , further comprising:
obtaining a plurality of excluded objects associated with the classification;
detecting the real-world object depicted in the video using an object recognition process; and
comparing the detected object depicted in the video to the plurality of expected excluded objects.

9. The method of claim 8 , further comprising: based on the comparing, identifying a given excluded object from the plurality of excluded objects that is excluded from the detected real-world object; and searching for an AR item corresponding to the given excluded object.

10. The method of claim 9 , further comprising: generating a three-dimensional (3D) mesh representation of the real-world environment; obtaining a plurality of real-world items corresponding to the classification; detecting that the plurality of real-world items excludes the detected object depicted in the video; determining, based on the 3D mesh representation, that physical space is available for a given one of the plurality of real-world items; and selecting an AR item corresponding to the given one of the plurality of real-world items.

11. The method of claim 1 , wherein the classification corresponds to a kitchen, the method further comprising: detecting that the real-world object depicted in the video corresponds to a first type of kitchen appliance; and searching a plurality of AR items to identify an AR item corresponding to the first type of kitchen appliance, wherein the AR item corresponding to the first type of kitchen appliance represents a model of the first type of kitchen appliance that is different than the real-world object.

12. The method of claim 1 , wherein determining the classification for the real-world environment comprises:
comparing, by the trained neural network, the real-world object depicted in the video to a plurality of lists of expected objects, each list associated with a different real-world environment classification;
computing, for each list, a relevancy score based on a quantity or percentage of objects depicted in the video that match objects in the list;
identifying the list with the highest relevancy score; and
determining the classification for the real-world environment based on the real-world environment classification associated with the identified list having the highest relevancy score.

13. The method of claim 1 , further comprising: generating a three-dimensional (3D) mesh representation of the real-world environment; detecting that the real-world object depicted in the video fails to satisfy one or more fit parameters of the 3D mesh representation; identifying, based on the 3D mesh representation, a recommended item that satisfies the one or more fit parameters of the 3D mesh representation and is of a same type as the real-world object included in the video; and retrieving an AR item corresponding to the recommended item in response to identifying the recommended item.

14. The method of claim 1 , wherein determining the classification for the real-world environment further comprises:
applying, by the trained neural network, multiple classifiers to the video, wherein at least one classifier is trained to classify a room and provide an estimated age range associated with the room;
determining, based on the estimated age range, a specific room type classification;
generating, by each classifier, a classification score indicating an accuracy of the generated real-world environment classification; and
assigning the real-world environment classification based on the classification with the highest score among the multiple classifiers.

15. The method of claim 14 , wherein the specific room type classification is selected from a plurality of different types of bedrooms each associated with a different age range, further comprising modifying an orientation of the AR item b

Assignees

• Snap Inc

Inventors

• Assouline Avihay
• Berger Itamar
• Dudovitch Gal
• Harel Peleg
• Sasson Gal

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T17/205

  Re-meshing · CPC title

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/20

  in augmented reality scenes · CPC title