Marker-based shared augmented reality session creation

Patent metadata

Field	Value
Publication number	US-12198376-B2
Application number	US-202318144171-A
Country	US
Kind code	B2
Filing date	May 6, 2023
Priority date	Dec 27, 2019
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

Method for creating marker-based shared augmented reality (AR) session starts with initializing a shared AR session by a first device and by a second device. The first device displays on a display a marker. The second device detects the marker using a camera included in the second device and captures an image of the marker using the camera. The second device determines a transformation between the first device and the second device using the image of the marker. A common coordinate frame is then determined using the transformation, the shared AR session is generated using the common coordinate frame, and the shared AR session is caused to be displayed by the first device and by the second device. Other embodiments are described herein.

First claim

What is claimed is:

1. A method comprising:
initializing a shared augmented reality (AR) session by a first device and by a second device;
detecting a marker using a camera of the second device;
capturing a data stream of images of the marker using the camera;
determining a transformation between the first device and the second device using an image of the marker from the data stream of images of the marker, wherein determining the transformation comprises:
receiving, by the second device, a stream of first device transformations from the first device,
determining, by the second device, an alignment of the stream of first device transformations with the data stream of images of the marker, and
selecting, based on the alignment, the image of the marker from the data stream of images of the marker and a corresponding first device transformation from the stream of first device transformations;

determining a common coordinate frame using the transformation;
generating the shared AR session using the common coordinate frame; and
causing the shared AR session to be displayed by the first device and by the second device.

2. The method of claim 1 , wherein initializing the shared AR session by the first device and by the second device comprises:
determining, by the first device, a first device transformation, wherein the first device transformation is based on a first device current pose and a first device origin, wherein the first device origin is an origin of a coordinate frame tracked by the first device;
determining, by the second device, a second device transformation, wherein the second device transformation is based on a second device current pose and a second device origin, wherein the second device origin is an origin of a coordinate frame tracked by the second device.

3. The method of claim 2 , wherein determining the transformation between the first device and the second device using the image of the marker further comprises:
determining, by the second device, a second offset based on the first device transformation, wherein the second offset is an offset of the second device current pose and the first device origin.

4. The method of claim 3 , wherein determining the transformation between the first device and the second device using the image of the marker further comprises:
determining the transformation using a position of the marker in the image, a scale of the marker in the image, or a rotation of the marker in the image.

5. The method of claim 4 , further comprising:
performing, by the second device, a gravity check, wherein performing the gravity check includes:
determining whether a direction of gravity based on the first device transformation corresponds to a direction of gravity based on the transformation between the first device and the second device.

6. The method of claim 1 , wherein the marker includes a session identifier.

7. The method of claim 6 , further comprising:
extracting, by the second device, the session identifier from the image of the marker.

8. The method of claim 7 , further comprising:
accessing by the first device a communication session using the session identifier included in the marker; and
accessing, by the second device, the communication session using the session identifier that is extracted from the image of the marker.

9. The method of claim 1 , wherein the marker is a Quick Response (QR) code, a natural feature marker, or a user interface of an application displayed on a display of the first device, or the first device.

10. The method of claim 1 , wherein the marker is a rotating marker and the selecting of the image of the marker from the data stream of images of the marker is based on a rotation speed of the marker.

11. A system comprising:
a processor; and
a memory component having instructions stored thereon, when executed by the processor, causes the processor to perform operations comprising:
detecting a marker using a camera of a second device;
capturing a data stream of images of the marker using the camera;
determining a transformation between a first device and the second device using an image of the marker from the data stream of images of the marker, wherein determining the transformation comprises:
receiving from the first device, a stream of first device transformations,
determining an alignment of the stream of first device transformations with the data stream of images of the marker, and
selecting, based on the alignment, the image of the marker from the data stream of images of the marker and a corresponding first device transformation from the stream of first device transformations;

determining a common coordinate frame using the transformation;
generating a shared AR session using the common coordinate frame; and
causing the shared AR session to be displayed by the first device and by the second device.

12. The system of claim 11 , wherein the marker includes a session identifier.

13. The system of claim 12 , wherein the operations further comprising:
extracting the session identifier from the image of the marker.

14. The system of claim 13 , wherein the operations further comprising:
providing access to the first device to a communication session based on the session identifier included in the marker; and
providing access to the second device to the communication session based on the session identifier that is extracted from the image of the marker.

15. The system of claim 11 , wherein the marker is a Quick Response (QR) code, a natural feature marker, a user interface of an application displayed on a display of the first device, or the first device.

16. The system of claim 11 ,
wherein each first device transformation in the stream is based on a first device pose and a first device origin, wherein the first device origin is an origin of a coordinate frame tracked by the first device.

17. The system of claim 16 , wherein determining the transformation between the first device and the second device using the image of the marker further comprises:
determining a second offset based on a first device transformation, wherein the second offset is an offset of a second device current pose and the first device origin.

18. The system of claim 17 , wherein determining the transformation between the first device and the second device using the image of the marker further comprises:
determining the transformation using a position of the marker in the image, a scale of the marker in the image, or a rotation of the marker in the image.

19. A non-transitory computer-readable storage medium having stored thereon, instructions when executed by a processor, causes the processor to perform operations comprising:
detecting a marker using a camera of a second device;
capturing a data stream of images of the marker using the camera;
determining a transformation between a first device and the second device using an image of the marker from the data stream of images of the marker, wherein determining the transformation comprises:
receiving from the first device, a stream of first device transformations,
determining an alignment of the stream of first device transformations with the data stream of images of the marker, and
selecting, based on the alignment, the image of the marker from the data stream of images of the marker and a corresponding first device transformation from the stream of first device transformations;

determining a common coordinate frame using the transformation;
generating a shared AR session using the com

Assignees

• Snap Inc

Inventors

• Cowburn Piers
• Li David
• Müller Sandvik Isac Andreas
• Pan Qi
• Zohar Matan

Classifications

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G06T2219/004

  Annotating, labelling · CPC title

• G06T2207/30204

  Marker · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title