Systems and methods for generating multi-user augmented reality content

Patent metadata

Field	Value
Publication number	US-11847716-B2
Application number	US-202217982669-A
Country	US
Kind code	B2
Filing date	Nov 8, 2022
Priority date	Jul 27, 2020
Publication date	Dec 19, 2023
Grant date	Dec 19, 2023

Abstract

Systems and methods are provided for the generation of augmented reality (AR) content that provides a shared AR experience involving multiple users. Shared AR experiences can improve the communication and collaboration between multiple simultaneous users. According to an embodiment, AR content is generated for a first user in a shared AR experience. The AR content includes at least one of a render of a model, a virtual representation of a second user in the shared AR experience, a virtual representation of a user interaction in the shared AR experience, and spatial audio content. Modifications to a shared AR experience are also provided. These modifications may be initiated based on instructions from one user and be reflected in the AR content generated for multiple users.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
obtaining a three-dimensional (3D) model;
determining, based on first information obtained by a first device, a position associated with a first user within a virtual coordinate system;
determining, based on second information obtained by a second device, a position associated with a second user within the virtual coordinate system, the second user remote from the first user;
obtaining, from the second device, audio of the second user;
generating augmented reality (AR) content, the AR content comprising:
a render of the 3D model based on the position associated with the first user within the virtual coordinate system,
a virtual representation of the second user based on the position associated with the second user relative to the position associated with the first user within the virtual coordinate system, and
audio content based on the audio of the second user, the audio content having a directionality corresponding to the position associated with the second user relative to the position associated with the first user within the virtual coordinate system; and

outputting the AR content for presentation at the first device, wherein the first device is to present the render of the 3D model and the virtual representation of the second user overlaid onto a view of a real-world space associated with the first user.

2. The computer-implemented method of claim 1 , further comprising:
obtaining, from the second device, an image of the second user, and wherein the virtual representation of the second user for presentation at the first device includes at least a portion of the image of the second user.

3. The computer-implemented method of claim 2 , wherein audio content is obtained from a microphone of the second device and the image of the second user is obtained from a front-facing camera of the second device.

4. The computer-implemented method of claim 1 , wherein the 3D model is a virtual model of a product for sale.

5. The computer-implemented method of claim 1 , wherein the product is for sale online by the second user.

6. The computer-implemented method of claim 1 , wherein:
the first information comprises a representation of the real-world space; and
determining the position associated with the first user within the virtual coordinate system comprises:
obtaining a position of the 3D model within the real-world space,
mapping the virtual coordinate system to the real-world space based on the position of the 3D model within the real-world space, and
determining a position of the first user within the real-world space, wherein the position of the first user within the virtual coordinate system is based on the position of the first user within the real-world space.

7. The computer-implemented method of claim 6 , wherein determining the position of the first user within the real-world space comprises determining, based on the representation of the real-world space, a distance from a surface in the real-world space to the first user.

8. The computer-implemented method of claim 7 , wherein determining the position of the first user within the real-world space comprises determining a position of the first device within the real-world space.

9. The computer-implemented method of claim 6 , wherein:
the representation of the real-world space comprises an image of the real-world space captured by the first device; and
determining the position of the first user within the real-world space is based on image analysis of the image.

10. The computer-implemented method of claim 6 , wherein:
the real-world space is a first space;
the second information comprises a representation of a second space, the second space being different from the first space; and
obtaining the position associated with the second user within the virtual coordinate system comprises:
obtaining a position of the 3D model within the second space,
mapping the virtual coordinate system to the second space based on the position of the 3D model within the second space, and
determining a position of the second user within the second space, wherein the position of the second user within the virtual coordinate system is based on position of the second user within the second space.

11. A system comprising:
at least one processor; and
a memory storing processor-executable instructions that, when executed, cause the at least one processor to:
obtain a three-dimensional (3D) model;
determine, based on first information obtained by a first device, a position associated with a first user within a virtual coordinate system;
determine, based on second information obtained by a second device, a position associated with a second user within the virtual coordinate system, the second user remote from the first user;

obtain, from the second device, audio of the second user;
generate augmented reality (AR) content, the AR content comprising:
a render of the 3D model based on the position associated with the first user within the virtual coordinate system,
a virtual representation of the second user based on the position associated with the second user relative to the position associated with the first user within the virtual coordinate system, and
audio content based on the audio of the second user, the audio content having a directionality corresponding to the position associated with the second user relative to the position associated with the first user within the virtual coordinate system; and
output the AR content for presentation at the first device, wherein the first device is to present the render of the 3D model and the virtual representation of the second user overlaid onto a view of a real-world space associated with the first user.

12. The system of claim 11 , wherein the at least one processor is further to:
obtain, from the second device, an image of the second user, and wherein the virtual representation of the second user for presentation at the first device includes at least a portion of the image of the second user.

13. The system of claim 12 , wherein audio content is obtained from a microphone of the second device and the image of the second user is obtained from a front-facing camera of the second device.

14. The system of claim 11 , wherein the 3D model is a virtual model of a product for sale.

15. The system of claim 11 , wherein the product is for sale online by the second user.

16. The system of claim 11 , wherein:
the first information comprises a representation of the real-world space; and
determining the position associated with the first user within the virtual coordinate system comprises:
obtaining a position of the 3D model within the real-world space,
mapping the virtual coordinate system to the real-world space based on the position of the 3D model within the real-world space, and
determining a position of the first user within the real-world space, wherein the position of the first user within the virtual coordinate system is based on the position of the first user within the real-world space.

17. The system of claim 16 , wherein determining the position of the first user within the real-world space comprises determining, based on the representation of the real-world space, a distance from a surface in the real-world space to the first user.

18. The system of claim 17 , wherein determining the position of the first user within the real-world space comprises determining a position of the first device within the real-world space.

19. The syst

Assignees

• Shopify Inc

Inventors

• Haapoja Juho Mikko
• Delgado Byron Leonel
• Leroux Stephan
• Beauchamp Daniel

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/16

  Sound input; Sound output (speech processing G10L) · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06Q10/0833

  Tracking · CPC title