Systems and methods for modeling real-world objects in virtual scenes

Patent metadata

Field	Value
Publication number	US-12346948-B2
Application number	US-202217950225-A
Country	US
Kind code	B2
Filing date	Sep 22, 2022
Priority date	Aug 5, 2022
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

A computer-implemented method includes: obtaining a three-dimensional (3D) representation of a first real-world environment; identifying a real-world object of interest in a second real-world environment, the first real-world environment different from the second real-world environment; determining a first position in the 3D representation of the first real-world environment corresponding to the real-world object of interest; and generating an augmented reality (AR) version of the first real-world environment for presentation in the second real-world environment using the 3D representation of the first real-world environment and based on positioning the real-world object of interest in the first position in the AR version of the first real-world environment.

First claim

The invention claimed is:

1. A computer-implemented method, comprising:
obtaining a three-dimensional (3D) representation of a first real-world environment;
identifying a real-world object of interest in a second real-world environment, the first real-world environment different from the second real-world environment;
determining a first position in the 3D representation of the first real-world environment corresponding to the real-world object of interest based on determining a position of a similar real-world object in the first real-world environment; and
generating an augmented reality (AR) version of the first real-world environment for presentation in the second real-world environment using the 3D representation of the first real-world environment and based on positioning the real-world object of interest at the position of the similar real-world object in the AR version of the first real-world environment so as to replace the similar real-world object.

2. The method of claim 1 , wherein generating the AR version of the first real-world environment comprises removing the similar real-world object from the 3D representation of the first real-world environment.

3. The method of claim 1 , wherein generating the AR version of the first real-world environment comprises:
determining, based on the 3D representation, that the first real-world environment does not contain a similar real-world object; and
in response, positioning the real-world object of interest within an empty space that is sized to fit the real-world object of interest in the AR version of the first real-world environment.

4. The method of claim 3 , wherein determining the first position in the 3D representation of the first real-world environment comprises determining positions of one or more objects in the 3D representation of the first real-world environment.

5. The method of claim 4 , wherein determining the first position in the 3D representation of the first real-world environment comprises determining a position of a second object in the 3D representation of the first real-world environment, the second object satisfying a defined condition with respect to the real-world object of interest.

6. The method of claim 1 , wherein the 3D representation of the first real-world environment comprises metadata indicating at least one of location or boundary associated with at least one object in the 3D representation.

7. The method of claim 1 , further comprising obtaining a first image of the real-world object of interest and wherein generating the AR version of the first real-world environment comprises combining the first image and the 3D representation of the first real-world environment.

8. The method of claim 1 , wherein the AR version of the first real-world environment is generated responsive to determining that a defined trigger condition is satisfied.

9. The method of claim 8 , wherein the defined trigger condition relates to at least one of:
a detected pose of a user relative to the real-world object of interest;
input of the user received via an input interface;
a distance of the user relative to the real-world object of interest; or
detected contact between the user and the real-world object of interest.

10. The method of claim 1 , wherein obtaining the 3D representation of the first real-world environment comprises obtaining 3D scan data including at least one of camera data or LiDAR sensor data.

11. The method of claim 1 , wherein generating the AR version of the first real-world environment comprises identifying a first subregion of a first image containing the real-world object of interest and a second subregion of the first image that does not contain the real-world object of interest.

12. The method of claim 11 , wherein generating the AR version of the first real-world environment comprises combining the 3D representation of the first real-world environment with the first image such that the second subregion of the first image is hidden in the AR version of the first real-world environment.

13. The method of claim 1 , further comprising:
obtaining depth data associated with the second real-world environment; and
partitioning an image of the second real-world environment using the depth data to obtain an image segment containing the real-world object,
wherein generating the AR version of the first real-world environment comprises combining the image segment with the 3D representation of the first real-world environment.

14. The method of claim 13 , wherein the depth data comprises a depth map of the second real-world environment generated using a 3D scanner.

15. The method of claim 14 , further comprising:
obtaining rotation and position data associated with the 3D scanner capturing the depth map; and
matching pixels of the depth map to locations in the image of the second real-world environment based on the rotation and position data.

16. The method of claim 14 , further comprising determining a bounding box representing a spatial extent of the real-world object in the second real-world environment, wherein the bounding box is determined based on the depth data.

17. A computing system, comprising:
a processor; and
a memory coupled to the processor, the memory storing processor-executable instructions that, when executed, are to cause the processor to:
obtain a three-dimensional (3D) representation of a first real-world environment;
identify a real-world object of interest in a second real-world environment, the first real-world environment different from the second real-world environment;
determine a first position in the 3D representation of the first real-world environment corresponding to the real-world object of interest based on determining a position of a similar real-world object in the first real-world environment; and
generate an augmented reality (AR) version of the first real-world environment for presentation in the second real-world environment using the 3D representation of the first real-world environment and based on positioning the real-world object of interest at the position of the similar real-world object in the AR version of the first real-world environment so as to replace the similar real-world object.

18. The computing system of claim 17 , wherein the instructions, when executed, are to further cause the processor to:
obtain depth data associated with the second real-world environment; and
partition an image of the second real-world environment using the depth data to obtain an image segment containing the real-world object, wherein generating the AR version of the first real-world environment comprises combining the image segment with the 3D representation of the first real-world environment.

19. The computing system of claim 17 , wherein generating the AR version of the first real-world environment comprises:
determining, based on the 3D representation, that the first real-world environment does not contain a similar real-world object; and
in response, positioning the real-world object of interest within an empty space that is sized to fit the real-world object of interest in the AR version of the first real-world environment.

20. A non-transitory processor-readable medium storing processor-executable instructions that, when executed by a processor, are to cause the processor to:
obtain a three-dimensional (3D) representation of a first real-world environment;
identify a real-world object of interest in a second real-world environment, the first real-world environment differen

Assignees

• Shopify Inc

Inventors

• Maschmeyer Russ
• Florenzano Eric Andrew
• Letkeman Brennan
• Bello Diego Macario
• Beauchamp Daniel

Classifications

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2210/04

  Architectural design, interior design · CPC title

• G06Q30/0627 Primary

  by specifying product or service characteristics, e.g. product dimensions · CPC title