AR item placement in a video

Patent metadata

Field	Value
Publication number	US-12499626-B2
Application number	US-202117566032-A
Country	US
Kind code	B2
Filing date	Dec 30, 2021
Priority date	Dec 30, 2021
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth information related to the real-world environment; and generating a 3D model of the real-world environment. The operations further include determining 3D placement and orientation for an AR item based on data associated with the AR item and the 3D model of the real-world environment and causing display of a marker in the video that specifies the 3D placement and orientation of the AR item. The operations further include rendering a display of the AR item within the video according to the 3D placement and orientation in response to movement of the marker within the video.

First claim

What is claimed is:

1 . A method comprising:
receiving, by one or more processors, a video that includes a depiction of one or more real-world objects in a real-world environment; obtaining depth data related to the real-world environment; generating a three-dimensional (3D) model of the real-world environment based on the video and the depth data; determining 3D placement and orientation for an augmented reality (AR) item based on data associated with the AR item and the 3D model of the real-world environment; causing display of a graphic in the video that specifies the 3D placement and orientation of the AR item; rendering a display of the AR item within the video according to the 3D placement and orientation in response to movement of the graphic within the video; determining from the 3D model that a real-world television is currently placed on a wall; in response to determining that the real-world television is currently placed on the wall, selecting an additional AR item comprising a television stand; and selecting a floor of the real-world environment as the 3D placement and orientation for the AR item representing the television and the additional AR item comprising the television stand.

2 . The method of claim 1 , further comprising:
overlaying the graphic on the video at a first 3D placement and orientation in the real-world environment; determining that the first 3D placement and orientation fails to match placement and orientation parameters associated with the AR item; and in response to determining that the first 3D placement and orientation fails to match the placement and orientation parameters associated with the AR item, maintaining the display of the graphic.

3 . The method of claim 2 , further comprising:
receiving input that moves the graphic to a second 3D placement and orientation in the real-world environment; determining that the second 3D placement and orientation matches the placement and orientation parameters associated with the AR item; and in response to determining that the second 3D placement and orientation matches the placement and orientation parameters associated with the AR item, replacing the display of the graphic with the display of the AR item.

4 . The method of claim 3 , wherein the placement and orientation parameters specify fit data for the AR item, and wherein the second 3D placement and orientation comprises available free space that corresponds to the fit data, further comprising:
maintaining a display of the graphic at a center of a display of the video; and moving a camera that is capturing the video to move the graphic at the center of the display to the second 3D placement and orientation in the real-world environment.

5 . The method of claim 1 , further comprising:
receiving LiDAR sensor data to obtain the depth data; processing the 3D model to identify the one or more real-world objects depicted in the video, wherein the 3D placement and orientation is determined based on the identified one or more real-world objects; identifying a plurality of placement and orientation positions for the AR item based on the data; and selecting a first placement and orientation position of the plurality of placement and orientation positions as the 3D placement and orientation based on the one or more real-world objects in the real-world environment.

6 . The method of claim 5 , further comprising:
identifying the one or more real-world objects in the real-world environment; and determining that a given one of the one or more real-world objects matches one or more parameters associated with the first placement and orientation position of the plurality of placement and orientation positions of the AR item.

7 . The method of claim 5 , further comprising:
obtaining priority values for the plurality of placement and orientation positions, a first priority value of the priority values associated with the first placement and orientation position being lower than a second priority value of the priority values associated with a second placement and orientation position of the plurality of placement and orientation positions; and searching the one or more real-world objects in the real-world environment for a first real-world object that matches the real-world object associated with the second placement and orientation position.

8 . The method of claim 7 , further comprising:
determining that the second placement and orientation position is unavailable; and in response to determining that the second placement and orientation position is unavailable, selecting the first placement and orientation position as the 3D placement and orientation instead of the second placement and orientation position.

9 . The method of claim 8 , further comprising:
detecting presence of an interfering real-world object depicted in the video at the second placement and orientation position, the second placement and orientation position being unavailable is determined in response to detecting the presence of the interfering real-world object.

10 . The method of claim 1 , further comprising:
determining that the data associated with the AR item specifies the wall as a placement position with a highest priority value.

11 . The method of claim 10 , further comprising combining the AR item representing the television with the additional AR item comprising the television stand to generate a combined AR item.

12 . The method of claim 11 , further comprising maintaining a display position of the combined AR item within the video as a camera that is capturing the video in real time is moved around the real-world environment.

13 . The method of claim 10 , wherein the additional AR item is selected in response to determining that the one or more real-world objects fail to include the television stand, and wherein the additional AR item is selected based on dimensions of the floor in the 3D model of the real-world environment.

14 . The method of claim 1 , further comprising:
receiving input that selects a different size of the AR item; and scaling the AR item in place based on the input that selects the different size.

15 . The method of claim 1 , further comprising:
receiving interaction data relating to the AR item, the interaction data comprising movement of the AR item within 3D space of the video; repositioning the AR item within the video based on the interaction data; and in response to the repositioning, hiding one or more portions of the AR item behind a real-world object of the one or more real-world objects in response to detecting overlap between a position of the AR item and a position of the real-world object.

16 . The method of claim 1 , wherein the AR item represents an appliance, further comprising:
determining that the data associated with the AR item specifies a first real-world object as a placement position with a highest priority value; accessing the 3D model of the real-world environment to determine that a real-world appliance is currently placed on the first real-world object; and in response to determining that the real-world appliance is currently placed on the first real-world object, selecting a second real-world object of the real-world environment as the 3D placement and orientation for the AR item representing the appliance.

17 . The method of claim 1 , further comprising training a neural network classifier to determine a real-world environment classification, wherein the real-world environment classification is used to select the 3D placement and orientation for

Assignees

• Snap Inc

Inventors

• Assouline Avihay
• Berger Itamar
• Dudovitch Gal
• Harel Peleg
• Sasson Gal

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/521

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title