Extended reality tracking using shared pose data

Patent metadata

Field	Value
Publication number	US-12482131-B2
Application number	US-202318450137-A
Country	US
Kind code	B2
Filing date	Aug 15, 2023
Priority date	Jul 10, 2023
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

Examples disclosed herein relate to the use of shared pose data in extended reality (XR) tracking. A communication link is established between a first XR device and a second XR device. The second XR device is worn by a user. The first XR device receives pose data of the second XR device via the communication link and captures an image of the user. The user is identified based on the image and the pose data.

First claim

What is claimed is:

1 . A method performed by a first extended reality (XR) device, the method comprising:
establishing a communication link between the first XR device and a second XR device that is worn by a user; receiving, via the communication link, pose data of the second XR device; capturing an image of the user; and identifying the user based on the image and the pose data of the second XR device, wherein the identifying of the user comprises:
projecting at least a subset of the pose data of the second XR device onto the image to obtain at least one two-dimensional (2D) projected position of the second XR device; and
determining that the user depicted in the image corresponds to the at least one 2D projected position of the second XR device.

2 . The method of claim 1 , wherein the establishing of the communication link comprises establishing a pose sharing session that enables the first XR device to track a pose of the second XR device based on the pose data, and the pose data is updated during the pose sharing session to reflect changes in the pose of the second XR device over time.

3 . The method of claim 2 , further comprising, subsequent to identifying the user:
determining, based on the tracking of the pose of the second XR device, that the user has exited a camera field of view of the first XR device.

4 . The method of claim 3 , further comprising, subsequent to determining that the user has exited the camera field of view:
determining, based on the tracking of the pose of the second XR device, that the user has re-entered the camera field of view.

5 . The method of claim 1 , further comprising:
based on the identifying of the user, rendering, based on the image and the pose data, an augmentation with respect to the user, wherein the augmentation is uniquely rendered for and associated with the user; and causing presentation of the augmentation on a display of the first XR device.

6 . The method of claim 5 , wherein the user is a second user, and wherein causing the presentation of the augmentation on the display of the first XR device comprises causing the augmentation to appear at least partially overlaid on the second user from a viewing perspective of a first user wearing the first XR device.

7 . The method of claim 5 , wherein the image is a first image, and wherein the method further comprises, subsequent to the presentation of the augmentation on the display of the first XR device:
determining, based on the pose data, that the user has exited and re-entered a camera field of view of the XR device; capturing a second image of the user; and re-identifying the user by matching the pose data of the second XR device with the user in the second image.

8 . The method of claim 7 , further comprising:
based on re-identifying the user:
identifying the augmentation associated with the user, and
re-rendering the augmentation with respect to the user.

9 . The method of claim 2 , wherein the pose of the second XR device comprises a position and orientation of the second XR device expressed in six degrees of freedom.

10 . The method of claim 1 , wherein the pose data comprises a plurality of poses generated by a Simultaneous Localization and Mapping (SLAM) system of the second XR device at different points in time.

11 . The method of claim 1 , wherein the image is a first image, and the method further comprises:
capturing a plurality of additional images of the user; and determining, based on the first image, the plurality of additional images, and the pose data, a trajectory of the second XR device.

12 . The method of claim 1 , further comprising, prior to identifying the user:
aligning a spatial reference system of the first XR device with a spatial reference system of the second XR device.

13 . The method of claim 12 , wherein aligning of the spatial reference system of the first XR device with the spatial reference system of the second XR device comprises scanning a common marker.

14 . The method of claim 13 , wherein aligning of the spatial reference system of the first XR device with the spatial reference system of the second XR device comprises ego-motion alignment.

15 . The method of claim 1 , further comprising:
generating, based on the image and the pose data, a body tracking prediction associated with the user.

16 . The method of claim 1 , wherein the communication link is a first communication link, and the method further comprises:
generating, based on the image and the pose data of the second XR device, a first landmark estimation for a detected body part of the user; establishing a second communication link between the first XR device and a third XR device, wherein the third XR device uses the pose data of the second XR device to generate a second landmark estimation for the detected body part; receiving, via the second communication link, the second landmark estimation and pose data of the third XR device; and processing the second landmark estimation and the pose data of the third XR device to adjust the first landmark estimation.

17 . The method of claim 2 , further comprising:
during the pose sharing session, transmitting, via the communication link, pose data of the first XR device to the second XR device.

18 . A first extended reality (XR) device comprising:
at least one memory that stores instructions; and at least one processor configured by the instructions to perform operations comprising:
establishing a communication link between the first XR device and a second XR device that is worn by a user;
receiving, via the communication link, pose data of the second XR device;
capturing an image of the user; and
identifying the user based on the image and the pose data of the second XR device, wherein the identifying of the user comprises:
projecting at least a subset of the pose data of the second XR device onto the image to obtain at least one two-dimensional (2D) projected position of the second XR device; and
determining that the user depicted in the image corresponds to the at least one 2D projected position of the second XR device.

19 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor of a first extended reality (XR) device, cause the at least one processor to perform operations comprising:
establishing a communication link between the first XR device and a second XR device that is worn by a user; receiving, via the communication link, pose data of the second XR device; capturing an image of the user; and identifying the user based on the image and the pose data of the second XR device, wherein the identifying of the user comprises:
projecting at least a subset of the pose data of the second XR device onto the image to obtain at least one two-dimensional (2D) projected position of the second XR device; and
determining that the user depicted in the image corresponds to the at least one 2D projected position of the second XR device.

20 . The first XR device of claim 18 , the operations further comprising:
based on the identifying of the user, rendering, based on the image and the pose data, an augmentation with respect to the user, wherein the augmentation is uniquely rendered for and associated with the user; and causing presentation of the augmentation on a display of the first XR device.

Assignees

• Snap Inc

Inventors

• Fulkerson Brian
• Muttenthaler Thomas
• Papandreou Georgios
• Wolf Daniel

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/20081

  Training; Learning · CPC title