Gesture-based shared AR session creation

Patent metadata

Field	Value
Publication number	US-11960651-B2
Application number	US-202016983693-A
Country	US
Kind code	B2
Filing date	Aug 3, 2020
Priority date	Mar 30, 2020
Publication date	Apr 16, 2024
Grant date	Apr 16, 2024

Abstract

Method of creating shared AR session based on a gesture starts with server receiving observed motion data from first device associated with first user. First device generating observed motion data based on an analysis of data stream comprising images of second user performing a gesture. Second user being associated with second device. Server receiving from second device captured motion data that corresponds to the gesture. Captured motion data being recorded by a sensor included in second device. Server determines whether there is a match between observed motion data from first device and captured motion data from second device. In response to determining that there is the match, server generates shared AR session between first device and second device and causes shared AR session to be displayed by first device and second device. Other embodiments are described herein.

First claim

What is claimed is:

1. A method comprising:
receiving, by a processor, an observed motion data from a first device associated with a first user, the first device generating the observed motion data based on an analysis of a data stream, the first device includes a camera that captures the data stream comprising video images of a second user performing a gesture as captured by the first user, the second user being associated with and coupled to a second device;
receiving from the second device a captured motion data that is recorded by a sensor included in the second device, wherein the sensor captures position, movement and orientation of the second device;
determining whether there is a match between the observed motion data from the first device and the captured motion data from the second device, wherein determining there is the match is based on the observed motion data and the captured motion data describing a same gesture being performed by the second user within a predetermined error threshold, wherein the gesture comprises a natural movement of the second user comprising a walking gait; and
in response to determining that there is the match,
generating a shared AR session between the first device and the second device; and
causing the shared AR session to be displayed by the first device and by the second device.

2. The method of claim 1 , wherein the shared AR session comprises a real-time messaging session.

3. The method of claim 1 , wherein the first device generating the observed motion data based on the analysis of the data stream further comprises:
the first device identifying positions of at least one body part of the second user and analyzing a motion of the at least one body part during the gesture.

4. The method of claim 3 , wherein the at least one body part includes an arm or hand of the second user.

5. The method of claim 1 , wherein the data stream further comprises images of a third user being associated with a third device, and wherein the first device identifies the second user in the data stream as being associated with the second device.

6. A system comprising:
a processor; and
a memory having instructions stored thereon, when executed by the processor, causes the system to perform operations comprising:
receiving an observed motion data from a first device associated with a first user, the first device generating the observed motion data based on an analysis of a data stream, the first device includes a camera that captures the data stream comprising video images of a second user performing a gesture as captured by the first user, the second user being associated with and coupled to a second device;
receiving from the second device a captured motion data that is recorded by a sensor included in the second device, wherein the sensor captures position, movement and orientation of the second device;
determining whether there is a match between the observed motion data from the first device and the captured motion data from the second device, wherein determining there is the match is based on the observed motion data and the captured motion data describing a same gesture being performed by the second user within a predetermined error threshold, wherein the gesture comprises a natural movement of the second user comprising a walking gait; and
in response to determining that there is the match,
generating a shared AR session between the first device and the second device; and
causing the shared AR session to be displayed by the first device and by the second device.

7. The system of claim 6 , wherein the shared AR session comprises a real-time messaging session.

8. The system of claim 6 , wherein the first device generating the observed motion data based on the analysis of the data stream further comprises:
the first device identifying positions of at least one body part of the second user and analyzing a motion of the at least one body part during the gesture.

9. The system of claim 8 , wherein the at least one body part includes an arm or hand of the second user.

10. The system of claim 6 , wherein the data stream further comprises images of a third user being associated with a third device, and wherein the first device identifies the second user in the data stream as being associated with the second device.

11. A non-transitory computer-readable storage medium having stored thereon instructions, when executed by a processor, causes the processor to perform operations comprising:
receiving an observed motion data from a first device associated with a first user, the first device generating the observed motion data based on an analysis of a data stream, the first device includes a camera that captures the data stream comprising video images of a second user performing a gesture as captured by the first user, the second user being associated with and coupled to a second device;
receiving from the second device a captured motion data that is recorded by a sensor included in the second device, wherein the sensor captures position, movement and orientation of the second device;
determining whether there is a match between the observed motion data from the first device and the captured motion data from the second device, wherein determining there is the match is based on the observed motion data and the captured motion data describing a same gesture being performed by the second user within a predetermined error threshold, wherein the gesture comprises a natural movement of the second user comprising a walking gait; and
in response to determining that there is the match,
generating a shared AR session between the first device and the second device; and
causing the shared AR session to be displayed by the first device and by the second device.

12. A method comprising:
capturing, using a camera included in a first device, a video of a user of a second device performing a gesture as captured by a first user of the first device, wherein the user of the second device is coupled to the second device;
generating, by a processor included in the first device, an observed motion data based on an analysis of the video of the user of the second device;
receiving, by the first device from the second device, a captured motion data that is recorded by a sensor included in the second device, wherein the sensor captures position, movement and orientation of the second device;
determining, by the first device, whether there is a match between the observed motion data and the captured motion data from the second device, wherein determining there is the match is based on the observed motion data and the captured motion data describing a same gesture being performed by the user of the second device within a predetermined error threshold, wherein the gesture comprises a natural movement of the second user comprising a walking gait; and
in response to determining that there is the match, transmitting a request to a server to generate a shared AR session between the first device and the second device.

13. The method of claim 12 , further comprising:
in response to determining that there is the match,
transmitting the request to the server to cause the shared AR session to be displayed by the first device and by the second device.

14. The method of claim 12 , wherein a shared AR session is established between the first device and the second device.

15. The method of claim 14 , further comprising:
in response to determining that there is the match,
identifying by the first device, the user of the second device in the video as being associated with the second device.

16

Assignees

• Snap Inc

Inventors

• Cowburn Piers George
• Li David
• Müller Sandvik Isac Andreas
• Pan Qi

Classifications

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04L51/04

  Real-time or near real-time messaging, e.g. instant messaging [IM] · CPC title