Build and update a virtual store based on a physical store

Patent metadata

Field	Value
Publication number	US-12488384-B2
Application number	US-202117332775-A
Country	US
Kind code	B2
Filing date	May 27, 2021
Priority date	May 27, 2021
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Disclosed herein are methods and systems for building and updating a virtual store based on a physical store. A computing system is structured to receive captured images of a physical space having a set of products. The physical space is simulated by a three-dimensional (3D) model of the physical space. A corresponding representation of the set of products is positioned relative to the 3D model. The computing system is structured to detect, based on the captured images, product information for products in the physical space and generate, based on the product information, an updated representation of the set of products based on the product information. The updated representation of the set of products corresponds to the products in the physical space as shown in the captured images.

First claim

What is claimed is:

1 . A method comprising:
receiving, by a computer, captured images of a physical space having a set of objects wherein the physical space is simulated by a three-dimensional (3D) model of the physical space and a corresponding representation of the set of objects positioned relative to the 3D model; determining, by the computer based on the captured images, information for objects in the physical space, the determining including:
segmenting, based on at least one visual attribute, the captured images into a plurality of portions,
extracting at least a portion of the captured images, and
identifying the objects based on the extracted at least the portion of the captured images,
wherein when an identity of a first object is not determined using a first captured image, the computer identifies the first object using a second captured image that depicts the first object; and
generating, by the computer based on the object information, an updated representation of the set of objects in the 3D model based on the object information, wherein the updated representation of the set of objects corresponds to the objects in the physical space as shown in the captured images.

2 . The method of claim 1 , further comprising updating the 3D model of the physical space based on the captured images.

3 . The method of claim 1 , wherein the 3D model of the physical space further comprises a surface rendering representative of a surface in the physical space as shown in the captured images.

4 . The method of claim 1 , wherein the object information comprises a 3D rendering of an object from the set of objects.

5 . The method of claim 4 , further comprising anchoring, by the computer, the updated representation of the object to the surface, comprising generating an object index comprising an object identifier item and a positioning coordinates item.

6 . The method of claim 5 , further comprising:
recognizing, by the computer, the object in the captured images; obtaining, by the computer, a 3D rendering of the object from a database; and generating the object index, comprising populating the object index with the object identifier that corresponds to the 3D rendering of the object from the database, wherein the 3D model of the physical space contains the 3D rendering of the object from the database as the updated representation of the object in a position corresponding to the positioning coordinates item from the object index.

7 . The method of claim 6 , wherein the 3D rendering of the object has a higher resolution than the captured images.

8 . The method of claim 5 , further comprising:
determining, by the computer, a location of the object; generating the positioning coordinates item for the object index based on the location; and rendering, by the computer, the object within the 3D model of the physical space to correspond to the location of the object in the physical space.

9 . The method of claim 8 , further comprising:
receiving, by the computer from a secondary device, positioning data representing the location of the object in the physical space; and determining, by the computer, a physical location of the object based on the positioning data.

10 . A non-transitory machine-readable storage medium having computer-executable instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, by a computer, captured images of a physical space having a set of objects wherein the physical space is simulated by a three-dimensional (3D) model of the physical space and a corresponding representation of the set of objects positioned relative to the 3D model; determining, by the computer based on the captured images, object information for objects in the physical space, the determining including:
segmenting, based on at least one visual attribute, the captured images into a plurality of portions,
extracting at least a portion of the captured images, and
identifying the objects based on the extracted at least the portion of the captured images,
wherein when an identity of a first object is not determined using a first captured image, the computer identifies the first object using a second captured image that depicts the first object; and
generating, by the computer based on the object information, an updated representation of the set of objects based on the object information, wherein the updated representation of the set of objects corresponds to the objects in the physical space as shown in the captured images.

11 . The medium of claim 10 , the operations further comprising updating the 3D model of the physical space based on the captured images.

12 . The medium of claim 10 , wherein the 3D model of the physical space further comprises a surface rendering representative of a surface in the physical space as shown in the captured images.

13 . The medium of claim 10 , wherein the object information comprises a 3D rendering of an object from the set of objects.

14 . The medium of claim 13 , the operations further comprising anchoring, by the computer, the updated representation of the object to the surface at least by generating an object index comprising an object identifier item and a positioning coordinates item.

15 . The medium of claim 14 , the operations further comprising:
recognizing, by the computer, the object in the captured images; obtaining, by the computer, a 3D rendering of the object from a database; and generating the object index, comprising populating the object index with the object identifier that corresponds to the 3D rendering of the object from the database, wherein the 3D model of the physical space contains the 3D rendering of the object from the database as the updated representation of the object in a position corresponding to the positioning coordinates item from the object index.

16 . The medium of claim 15 , wherein the 3D rendering of the object has a higher resolution than the captured images.

17 . The medium of claim 16 , the operations further comprising:
determining, by the computer, a location of the object; generating the positioning coordinates item for the object index based on the location; and rendering, by the computer, the object within the 3D model of the physical space to correspond to the location of the object in the physical space.

18 . The medium of claim 17 , the operations further comprising:
receiving, by the computer from a secondary device, positioning data representing the location of the object in the physical space; and determining, by the computer, a physical location of the object based on the positioning data.

19 . A computer system comprising a memory and at least one processor, the memory having computer-executable instructions stored thereon that cause the at least one processor to perform operations comprising:
receiving, by a computer, captured images of a physical space having a set of objects wherein the physical space is simulated by a three-dimensional (3D) model of the physical space and a corresponding representation of the set of objects positioned relative to the 3D model; determining, by the computer based on the captured images, object information for objects in the physical space, the determining including:
segmenting, based on at least one visual attribute, the captured images into a plurality of portions,
extracting at least a portion of the captured images, and
identifying the objects based o

Assignees

• Shopify Inc

Inventors

• Haapoja Juho Mikko
• Delgado Byron Leonel
• Leroux Stephan

Classifications

• G06Q30/0639

  Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title