Egocentric human body pose tracking

Patent metadata

Field	Value
Publication number	US-12307006-B2
Application number	US-202318368427-A
Country	US
Kind code	B2
Filing date	Sep 14, 2023
Priority date	Sep 15, 2022
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

A pose tracking system is provided. The pose tracking system includes an EMF tracking system having a user-worn head-mounted EMF source and one or more user-worn EMF tracking sensors attached to the wrists of the user. The EMF source is associated with a VIO tracking system such as AR glasses or the like. The pose tracking system determines a pose of the user's head and a ground plane using the VIO tracking system and a pose of the user's hands using the EMF tracking system to determine a full-body pose for the user. Metal interference with the EMF tracking system is minimized using an IMU mounted with the EMF tracking sensors. Long term drift in the IMU and the VIO tracking system are minimized using the EMF tracking system.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining, by one or more processors, using a first Electromagnetic Field (EMF) tracking sensor, first EMF tracking data of a first wrist of a user;
determining, by the one or more processors, using a second EMF tracking sensor, second EMF tracking data of a second wrist of the user;
determining, by the one or more processors, using a Visual Inertial Odometry (VIO) tracking system, VIO tracking data of a head of the user;
determining, by the one or more processors, head pose data of the head of the user based on the VIO tracking data;
determining, by the one or more processors, first wrist pose data of the first wrist using the first EMF tracking data;
determining, by the one or more processors, second wrist pose data of the second wrist using the second EMF tracking data;
mapping, by the one or more processors, the head pose data to a first joint in a 3D body model, the first wrist pose data to a second joint of the 3D body model, and the second wrist pose data to a third joint of the 3D body model;
generating, by the one or more processors, full 3D body model data of the user based on the first joint, the second joint, the third joint, and an Inverse Kinematic (IK) model, the full 3D body model data including a full body pose of the user; and
communicating, by the one or more processors, the 3D body model data to an extended Reality (XR) application for use in an XR user interface for the user.

2. The computer-implemented method of claim 1 , wherein determining the head and wrist pose data further comprises:
determining Inertial Measurement Unit (IMU) tracking data of the first EMF tracking sensor and the second EMF tracking sensor;
detecting interference in the EMF tracking data based on the EMF tracking data; and
correcting the EMF tracking data based on the IMU tracking data.

3. The computer-implemented method of claim 2 , further comprising:
determining IMU tracking data of the first EMF tracking sensor and the second EMF tracking sensor; and
correcting long-term drift in the IMU tracking data using the EMF tracking data.

4. The computer-implemented method of claim 1 , wherein an EMF tracking system includes the first EMF tracking sensor, the second EMF tracking sensor, and a head-mounted EMF source in a fixed relationship to the VIO tracking system, and wherein the first wrist pose data and the second wrist pose data is determined based on a pose of the VIO tracking system and a relative pose of the first EMF tracking sensor and the second EMF tracking sensor.

5. The computer-implemented method of claim 1 , further comprising:
determining, by the one or more processors, ground plane data based on the VIO tracking data; and
generating the 3D body model data is further based on the ground plane data.

6. The computer-implemented method of claim 1 , further comprising:
correcting, by the one or more processors, the EMF tracking data using a previous EMF position tracking data history.

7. The computer-implemented method of claim 6 , wherein correcting the EMF tracking data comprises using a Machine Learning (ML) for time series forecasting.

8. A machine, comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
determining, using a first Electromagnetic Field (EMF) tracking sensor, first EMF tracking data of a first wrist of a user;
determining, using a second EMF tracking sensor, second EMF tracking data of a second wrist of the user;
determining, using a Visual Inertial Odometry (VIO)-tracking system, VIO tracking data of a head of the user;
determining head pose data of the head of the user based on the VIO tracking data;
determining first wrist pose data of the first wrist using the first EMF tracking data;
determining second wrist pose data of the second wrist using the second EMF tracking data;
mapping the head pose data to a first joint in a 3D body model, the first wrist pose data to a second joint of the 3D body model, and the second wrist pose data to a third joint of the 3D body model;
generating full 3D body model data of the user based on the head and wrist pose data the first joint, the second joint, the third joint, and an Inverse Kinematic (IK) model, the full 3D body model data including a full body pose of the user; and
communicating the 3D body model data to an extended Reality (XR) application for use in an XR user interface for the user.

9. The machine of claim 8 , wherein determining the head and wrist pose data further comprises:
determining Inertial Measurement Unit (IMU) tracking data of the first EMF tracking sensor and the second EMF tracking sensor;
detecting interference in the EMF tracking data based on the EMF tracking data; and
correcting the EMF tracking data based on the IMU tracking data.

10. The machine of claim 9 , wherein the operations further comprise:
determining IMU tracking data of the first EMF tracking sensor and the second EMF tracking sensor; and
correcting long-term drift in the IMU tracking data using the EMF tracking data.

11. The machine of claim 8 , wherein an EMF tracking system further includes the first EMF tracking sensor, the second EMF tracking sensor, and a head-mounted EMF source in a fixed relationship to the VIO tracking system, and wherein the first wrist pose data and the second wrist pose data is determined based on a pose of the VIO tracking system and a relative pose of the first EMF tracking sensor and the second EMF tracking sensor.

12. The machine of claim 8 , wherein the operations further comprise:
determining, by the one or more processors, ground plane data based on the VIO tracking data; and
generating the 3D body model data is further based on the ground plane data.

13. The machine of claim 8 , wherein the operations further comprise:
correcting, by the one or more processors, the EMF tracking data using a previous EMF position tracking data history.

14. The machine of claim 13 , wherein correcting the EMF tracking data comprises using a Machine Learning (ML) for time series forecasting.

15. A machine-storage medium storing instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
determining, using a first Electromagnetic Field (EMF) tracking sensor, first EMF tracking data of a first wrist of a user;
determining, using a second EMF tracking sensor, second EMF tracking data of a second wrist of the user;
determining, using a Visual Inertial Odometry (VIO)-tracking system, VIO tracking data of a head of the user;
determining head pose data of the head of the user based on the VIO tracking data;
determining first wrist pose data of the first wrist using the first EMF tracking data;
determining second wrist pose data of the second wrist using the second EMF tracking data;
mapping the head pose data to a first joint in a 3D body model, the first wrist pose data to a second joint of the 3D body model, and the second wrist pose data to a third joint of the 3D body model;
generating full 3D body model data of the user based on the head and wrist pose data the first joint, the second joint, the third joint, and an Inverse Kinematic (IK) model, the full 3D body model data including a full body pose of the user; and
communicating the 3D body model data to an extended Reality (XR) application for use in an XR user interface for the user.

16. The machine-storage medium of claim 15 , wherein determining the

Assignees

• Snap Inc

Inventors

• Arakawa Riku
• Krishnan Gorumkonda Gurunandan
• Nayar Shree K
• Zhou Bing

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title