Extended reality motion prediction using hand kinematics

Patent metadata

Field	Value
Publication number	US-2025383715-A1
Application number	US-202418741375-A
Country	US
Kind code	A1
Filing date	Jun 12, 2024
Priority date	Jun 12, 2024
Publication date	Dec 18, 2025
Grant date	—

Abstract

Examples in the present disclosure relate to the prediction of motion of a body part by an extended reality (XR) device. Tracking data is captured by one or more sensors associated with the XR device. The tracking data is processed to track the body part. Based on the tracking of the body part and a kinematic model of the body part, kinematic state tracking data is dynamically updated. The kinematic model and the kinematic state tracking data are used to generate a predicted future kinematic state of the body part. In some examples, operation of the XR device is controlled based on the predicted future kinematic state.

First claim

1 . A method performed by an extended reality (XR) device, the method comprising:
accessing tracking data captured via one or more sensors of a plurality of sensors associated with the XR device; processing the tracking data to track at least one body part; dynamically updating kinematic state tracking data based on the tracking of the at least one body part and a kinematic model of the at least one body part; using the kinematic model and the kinematic state tracking data to generate a predicted future kinematic state of the at least one body part; and controlling operation of the XR device based on the predicted future kinematic state by dynamically selecting, based on the predicted future kinematic state, a subset of the plurality of sensors for tracking of the at least one body part, wherein the plurality of sensors comprises a plurality of cameras of a multi-camera object tracking system of the XR device, and selection of the subset is based on at least one of:
a camera of the plurality of cameras predicted to have the at least one body part closest to a center of a field of view of the camera;
a camera of the plurality of cameras predicted to have a least occluded view of the at least one body part; or
a camera of the plurality of cameras predicted to have a clearest view of a predetermined portion of the at least one body part.

2 . The method of claim 1 , wherein the XR device is a head-wearable XR device, and the at least one body part comprises at least part of a hand of a user of the XR device.

3 . The method of claim 2 , wherein the tracking data is processed to track positions of a plurality of landmarks comprising a plurality of joints of the hand, and the kinematic model is applied by the XR device to describe joint positions and joint angles.

4 . The method of claim 3 , wherein dynamically updating the kinematic state tracking data comprises tracking, over time, at least one of linear velocity of one or more of the plurality of joints, angular velocity of one or more of the plurality of joints, linear acceleration of one or more of the plurality of joints, angular acceleration of one or more of the plurality of joints, linear jerk of one or more of the plurality of joints, or angular jerk of one or more of the plurality of joints.

5 . The method of claim 1 , wherein the kinematic state tracking data tracks a kinematic state of the at least one body part over time, and the kinematic state is defined using the kinematic model.

6 . The method of claim 5 , wherein the predicted future kinematic state is generated based on the kinematic state tracking data and motion constraints defined by the kinematic model.

7 . The method of claim 5 , wherein the predicted future kinematic state is generated at a first point in time to predict the kinematic state of the at least one body part at a second point in time, and the second point in time is less than 1 second from the first point in time.

8 . The method of claim 5 , wherein the predicted future kinematic state is generated at a first point in time to predict the kinematic state of the at least one body part at a second point in time, and the second point in time is less than 500 ms from the first point in time.

9 . The method of claim 5 , wherein the kinematic state comprises a six degrees-of-freedom (6DoF) pose of the at least one body part in a real-world environment.

10 . The method of claim 1 , wherein at least some of the kinematic state tracking data is generated or updated, using inverse kinematics, by fitting positions of a plurality of landmarks obtained from the tracking data to the kinematic model.

11 . The method of claim 1 , wherein controlling the operation of the XR device based on the predicted future kinematic state further comprises:
identifying, based on the predicted future kinematic state, a predicted user action; determining a device action corresponding to the predicted user action; and synchronizing the device action with occurrence of the predicted user action.

12 . The method of claim 11 , wherein the predicted user action comprises a user of the XR device performing a detectable gesture.

13 . The method of claim 12 , wherein the device action comprises a response to the detectable gesture.

14 . The method of claim 11 , wherein the device action comprises triggering rendering of virtual content for presentation to a user via the XR device.

15 . (canceled)

16 . The method of claim 1 , wherein the predicted future kinematic state comprises a predicted pose of the at least one body part, and the subset is further selected based at least partially on the predicted pose in relation to the field of view of each respective camera of the plurality of cameras.

17 . The method of claim 1 , wherein processing of the tracking data comprises executing a machine learning model that is trained to perform object tracking.

18 . (canceled)

19 . An extended reality (XR) device comprising:
at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the XR device to perform operations comprising: accessing tracking data captured via one or more sensors of a plurality of sensors associated with the XR device; processing the tracking data to track at least one body part; dynamically updating kinematic state tracking data based on the tracking of the at least one body part and a kinematic model of the at least one body part; using the kinematic model and the kinematic state tracking data to generate a predicted future kinematic state of the at least one body part; and controlling operation of the XR device based on the predicted future kinematic state by dynamically selecting, based on the predicted future kinematic state, a subset of the plurality of sensors for tracking of the at least one body part, wherein the plurality of sensors comprises a plurality of cameras of a multi-camera object tracking system of the XR device, and selection of the subset is based on at least one of:
a camera of the plurality of cameras predicted to have the at least one body part closest to a center of a field of view of the camera;
a camera of the plurality of cameras predicted to have a least occluded view of the at least one body part; or
a camera of the plurality of cameras predicted to have a clearest view of a predetermined portion of the at least one body part.

20 . A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising:
accessing tracking data captured via one or more sensors of a plurality of sensors associated with an extended reality (XR) device; processing the tracking data to track at least one body part; dynamically updating kinematic state tracking data based on the tracking of the at least one body part and a kinematic model of the at least one body part; using the kinematic model and the kinematic state tracking data to generate a predicted future kinematic state of the at least one body part; and controlling operation of the XR device based on the predicted future kinematic state by dynamically selecting, based on the predicted future kinematic state, a subset of the plurality of sensors for tracking of the at least one body part, wherein the plurality of sensors comprises a plurality of cameras of a multi-camera object tracking system of the XR device, and sele

Assignees

• Snap Inc

Inventors

• Diem Markus
• Muttenthaler Thomas
• Wolf Daniel

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title