Systems and methods for representing user interactions in multi-user augmented reality

Patent metadata

Field	Value
Publication number	US-11908159-B2
Application number	US-202016939280-A
Country	US
Kind code	B2
Filing date	Jul 27, 2020
Priority date	Jul 27, 2020
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Systems and methods are provided for the generation of augmented reality (AR) content that provides a shared AR experience involving multiple users. Shared AR experiences can improve the communication and collaboration between multiple simultaneous users. According to an embodiment, AR content is generated for a first user in a shared AR experience. The AR content includes at least one of a render of a model, a virtual representation of a second user in the shared AR experience, a virtual representation of a user interaction in the shared AR experience, and spatial audio content. Modifications to a shared AR experience are also provided. These modifications may be initiated based on instructions from one user and be reflected in the AR content generated for multiple users.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
obtaining a position and orientation of a hand of a first user, the hand interacting with virtual content presented at a first device, the virtual content comprising a first render of a virtual 3D (three-dimensional) model defined within a virtual coordinate system, the position and the orientation of the hand being defined within the virtual coordinate system;
generating augmented reality (AR) content for presentation at a second device overlaid onto a view of a real-world space, the AR content comprising both:
a second render of the virtual 3D model based on a position of a second user within the virtual coordinate system, the position of the second user being based on information obtained by the second device representing the real-world space, and
a virtual representation of the hand of the first user interacting with the second render of the virtual 3D model by displaying the position and orientation of the hand of the first user in relation to the second render of the virtual 3D model, the virtual representation and the second render of the virtual 3D model both overlaid onto the view of the real-world space; and

outputting the AR content for presentation at the second device.

2. The computer-implemented method of claim 1 , further comprising:
obtaining a position of the first user within the virtual coordinate system,
wherein the position of the hand is based on the position of the first user within the virtual coordinate system.

3. The computer-implemented method of claim 1 , further comprising:
obtaining, using the first device, a representation of a space,
wherein obtaining the position of the hand of the first user within the virtual coordinate system comprises:
determining a position of the virtual 3D model within the space,
mapping the virtual coordinate system to the space based on the position of the virtual 3D model within the space, and
determining a position of the hand of the first user within the space, wherein the position of the hand of the first user within the virtual coordinate system is based on the position of the hand of the first user within the space.

4. The computer-implemented method of claim 3 , wherein:
the representation of the space comprises an image of the space captured by the first device; and
determining the position of the hand of the first user within the space is based on image analysis of the image.

5. The computer-implemented method of claim 1 , wherein:
the AR content is first AR content;
the real-world space is a first real-word space; and
the virtual content comprises second AR content for presentation at the first device overlaid onto a view of a second real-world space.

6. A system comprising:
memory to store a position and orientation of a hand of a first user, the hand interacting with virtual content presented at a first device, the virtual content comprising a first render of a virtual three-dimensional (3D) model defined within a virtual coordinate system, the position and orientation of the hand being defined within the virtual coordinate system; and
at least one processor configured to:
generate augmented reality (AR) content for presentation at a second device overlaid onto a view of a real-world space, the AR content comprising both:
a second render of the virtual 3D model based on a position of a second user within the virtual coordinate system, the position of the second user being based on information obtained by the second device representing the real-world space, and
a virtual representation of the hand of the first user interacting with the second render of the virtual 3D model by displaying the position and orientation of the hand of the first user in relation to the second render of the virtual 3D, the virtual representation and the second render of the virtual 3D model both overlaid onto the view of the real-world space; and

output the AR content for presentation at the second device.

7. The system of claim 6 , wherein the at least one processor is further configured to obtain a position of the first user within the virtual coordinate system, wherein the position of the hand is based on the position of the first user within the virtual coordinate system.

8. The system of claim 6 , wherein the at least one processor is further configured to:
obtain, using the first device, a representation of a space,
determine a position of the virtual 3D model within the space,
map the virtual coordinate system to the space based on the position of the virtual 3D model within the space, and
determine a position of the hand of the first user within the space, wherein the position of the hand of the first user within the virtual coordinate system is based on the position of the hand of the first user within the space.

9. The system of claim 8 , wherein:
the representation of the space comprises an image of the space captured by the first device; and
the position of the hand of the first user within the space is based on image analysis of the image.

10. The system of claim 6 , wherein:
the AR content is first AR content;
the real-world space is a first real-word space; and
the virtual content comprises second AR content for presentation at the first device overlaid onto a view of a second real-world space.

11. A non-transitory computer readable medium storing computer executable instructions which, when executed by a computer, cause the computer to:
obtain a position and orientation of a hand of a first user, the hand interacting with virtual content presented at a first device, the virtual content comprising a first render of a virtual three-dimensional (3D) model defined within a virtual coordinate system, the position and orientation of the hand being defined within the virtual coordinate system;
generate augmented reality (AR) content for presentation at a second device overlaid onto a view of a real-world space, the AR content comprising both:
a second render of the virtual 3D model based on a position of a second user within the virtual coordinate system, the position of the second user being based on information obtained by the second device representing the real-world space, and
a virtual representation of the hand of the first user interacting with the second render of the virtual 3D model by displaying the position and orientation of the hand of the first user in relation to the second render of the virtual 3D model, the virtual representation and the second render of the virtual 3D model both overlaid onto the view of the real-world space; and

output the AR content for presentation at the second device.

12. The computer-implemented method of claim 1 , wherein the virtual content is overlaid onto a virtual reality (VR) environment.

13. The system of claim 6 , wherein the virtual content is overlaid onto a virtual reality (VR) environment.

14. The computer-implemented method of claim 3 , wherein the representation of the space is based on a previously generated representation of the space including measurements previously obtained.

15. The system of claim 8 , wherein the representation of the space is based on a previously generated representation of the space including measurements previously obtained.

16. The computer-implemented method of claim 3 , wherein:
the first user is wearing an AR-specific glove; and
determining the position of the hand of the first user within the space is based a position of the AR-specific glove in the space.

17. The system of claim 8 , wherein:
the first u

Assignees

• Shopify Inc

Inventors

• Haapoja Juho Mikko
• Delgado Byron Leonel
• Leroux Stephan
• Beauchamp Daniel

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06Q30/0633

  Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087) · CPC title

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title