Augmented reality (AR)-based management of manual assembly processes

Patent metadata

Field	Value
Publication number	US-12585248-B2
Application number	US-202318188657-A
Country	US
Kind code	B2
Filing date	Mar 23, 2023
Priority date	Jan 17, 2023
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

A computer-implemented is disclosed. The method includes: obtaining image data from a plurality of cameras providing views of a designated assembly zone; updating a three-dimensional model of an assembly of a structure based on the obtained image data; and determining a location of a first component of the structure based on the updated three-dimensional model of the assembly.

First claim

The invention claimed is:

1 . A computer-implemented method, comprising:
obtaining image data from a plurality of cameras providing views of a designated assembly zone; updating a three-dimensional model of an assembly of a structure based on the obtained image data; determining locations of one or more components of sub-assemblies of the assembly based on the updated three-dimensional model of the assembly; identifying assemblers associated with one or more assembly steps; determining relative position information indicating positions of the identified assemblers with respect to the locations of the components of the sub-assemblies; and determining, for each of one or more of the identified assemblers, a respective subset of all assembly instructions to assign to the assembler based on the relative position information.

2 . The method of claim 1 , wherein the plurality of cameras includes cameras associated with augmented reality devices that are detected to be in or within a vicinity of the assembly zone.

3 . The method of claim 1 , wherein the plurality of cameras includes one or more cameras that are statically positioned in or within a vicinity of the assembly zone.

4 . The method of claim 1 , wherein determining the locations of the one or more components of the sub-assemblies comprises:
identifying a first subregion of the assembly zone using the updated three-dimensional model; and performing image analysis of video feeds of one or more cameras providing a view of the first subregion.

5 . The method of claim 1 , further comprising:
receiving a request to locate a first component; and responsive to receiving the request, presenting, via an AR device associated with an assembler in the assembly zone, an indication of a location of the first component.

6 . The method of claim 5 , further comprising, before receiving a request to locate the first component:
obtaining subsequent image data from the plurality of cameras; based on the obtained subsequent image data, determining that the first component is not at the location; and updating the three-dimensional model to include an indicator of a change in location of the first component.

7 . The method of claim 5 , wherein the indication comprises a graphical indicator of the location of the component that is overlaid on a view of the real-world space.

8 . The method of claim 1 , wherein identifying the assemblers comprises detecting that an assembler has joined a first assembly process.

9 . The method of claim 8 , wherein detecting that an assembler has joined the first assembly process comprises detecting one or more of: input of a code; login to a shared project space; a defined object in a camera feed associated with the assembler; defined audio cues; or calibration by contact.

10 . The method of claim 1 , wherein identifying the assemblers comprises determining that one or more entities that are prompted to join a first assembly process based on their current locations have joined the first assembly process.

11 . The method of claim 1 , further comprising causing to be displayed, via an AR device associated with the assembler, virtual guidance corresponding to the respective subset of assembly instructions.

12 . The method of claim 1 , further comprising:
monitoring completion progress of the identified assemblers in completing their respective assigned assembly tasks; and based on the monitoring, determining subsequent assembly instructions for assigning to one or more of the identified assemblers.

13 . The method of claim 1 , further comprising:
identifying a first one of the assemblers to receive location information for the first components; and causing to be displayed, via an AR device associated with the first assembler, a graphical indication of the location of the first component.

14 . The method of claim 13 , wherein the first one of the assemblers is identified based on determining proximity of one or more of the identified assemblers to the location of the first component.

15 . The method of claim 13 , wherein the first one of the assemblers is identified based on determining an identity of an assembler to receive assembly instructions associated with the first component.

16 . The computing system of claim 15 , wherein the relative position information further indicates positions of the identified assemblers with respect to each other.

17 . The method of claim 1 , wherein the relative position information further indicates positions of the identified assemblers with respect to each other.

18 . A computing system, comprising:
a processor; and a memory coupled to the processor, the memory storing computer-executable instructions that, when executed by the processor, configure the processor to:
obtain image data from a plurality of cameras providing views of a designated assembly zone;
update a three-dimensional model of an assembly of a structure based on the obtained image data;
determine locations of one or more components of sub-assemblies of the assembly based on the updated three-dimensional model of the assembly;
identify assemblers associated with one or more assembly steps;
determine relative position information indicating positions of the identified assemblers with respect to the locations of the components of the sub-assemblies; and
determine, for each of one or more of the identified assemblers, a respective subset of all assembly instructions to assign to the assembler based on the relative position information.

19 . The computing system of claim 18 , wherein determining the location of a component of a sub-assembly comprises:
identifying a first subregion of the assembly zone using the updated three-dimensional model; and performing image analysis of video feeds of one or more cameras providing a view of the first subregion.

20 . A non-transitory processor-readable medium storing computer-executable instructions that, when executed by a processor, are to cause the processor to:
obtain image data from a plurality of cameras providing views of a designated assembly zone; update a three-dimensional model of an assembly of a structure based on the obtained image data; determine locations of one or more components of sub-assemblies of the assembly based on the updated three-dimensional model of the assembly; identify assemblers associated with one or more assembly steps; determine relative position information indicating positions of the identified assemblers with respect to the locations of the components of the sub-assemblies; and determine, for each of one or more of the identified assemblers, a respective subset of all assembly instructions to assign to the assembler based on the relative position information.

Assignees

• Shopify Inc

Inventors

• Maschmeyer Russ
• Florenzano Eric Andrew
• Letkeman Brennan
• Bello Diego Macario
• Beauchamp Daniel
• Padgett Neil Leonard
• Jayatunga Hettige Ray Perera
• Lepp James
• Ngo Andrew Nhat-Nam

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G05B2219/32014

  Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command · CPC title

• G05B19/41885

  characterised by modeling, simulation of the manufacturing system · CPC title

• G05B19/41805 Primary

  characterised by assembly · CPC title