Hand tracking based on wrist rotation and arm movement

Patent metadata

Field	Value
Publication number	US-2025036206-A1
Application number	US-202418761848-A
Country	US
Kind code	A1
Filing date	Jul 2, 2024
Priority date	Jul 24, 2023
Publication date	Jan 30, 2025
Grant date	—

Abstract

Various implementations track hand motion to interpret a scroll movement. For example, an example process may include obtaining sensor data associated with a hand via one or more sensors in a physical environment. The process may further include determining positional data corresponding to three-dimensional (3D) positions of two points on the hand based on the sensor data. The process may further include determining whether a movement of the hand includes a wrist rotation associated with the hand rotating about the wrist, an arm motion associated with movement of the arm, or a combination of the wrist rotation and the arm motion. The process may further include determining the movement of the hand as a user interaction event based on the positional data and whether the movement of the hand includes the wrist rotation, the arm motion, or a combination of the wrist rotation and the arm motion.

First claim

What is claimed is:

1 . A method comprising:
at a device having a processor and one or more sensors:
obtaining sensor data associated with a hand via the one or more sensors in a physical environment, the hand comprising a wrist and is associated with an arm;
determining positional data corresponding to three-dimensional (3D) positions of two points on the hand based on the sensor data;
based on the determined positional data, determining whether a movement of the hand comprises a wrist rotation associated with the hand rotating about the wrist, an arm motion associated with movement of the arm, or a combination of the wrist rotation and the arm motion; and
determining the movement of the hand as a user interaction event based on the positional data and whether the movement of the hand comprises the wrist rotation, the arm motion, or a combination of the wrist rotation and the arm motion.

2 . The method of claim 1 , wherein a sensitivity of determining the movement of the hand as the user interaction event is increased based on determining the movement of the hand comprises at least the wrist rotation.

3 . The method of claim 1 , wherein determining the movement of the hand based on the determined positional data comprises determining whether the movement of the hand is classified as a wrist-based motion or an arm-based motion.

4 . The method of claim 3 , wherein the movement of the hand is determined based on a pinch centroid about an arm pivot in response to determining that the movement of the hand is classified as an arm-based motion.

5 . The method of claim 3 , wherein the movement of the hand is determined based on a pinch centroid about an arm pivot and the pinch centroid about a wrist pivot in response to determining that the movement of the hand is classified as a wrist-based motion.

6 . The method of claim 1 , wherein determining whether the movement of the hand comprises wrist rotation comprises determining a type of motion based on an amount of wrist movement compared to a threshold.

7 . The method of claim 1 , wherein the user interaction event comprises a scrolling action.

8 . The method of claim 1 , wherein determining the movement of the hand is based on an angular change.

9 . The method of claim 1 , wherein a first point of the two points on the hand comprises a wrist joint, and a second point of the two points on the hand comprises a pinch centroid associated with at least two fingers of the hand.

10 . The method of claim 1 , wherein determining the positional data corresponding to the 3D positions of two points on the hand is initiated based on identifying a pinch action associated with two or more fingers of the hand.

11 . The method of claim 1 , wherein determining the movement of the hand as the user interaction event is initiated based on a first user action.

12 . The method of claim 1 , wherein determining the movement of the hand is stopped based on a second user action.

13 . The method of claim 1 , further comprising:
determining a velocity of the movement of the hand; and
updating the positional data corresponding to the 3D positions of the two points on the hand based on the velocity of the hand exceeding a threshold.

14 . The method of claim 1 , further comprising:
obtaining a second sensor data signal associated with a pose of a head or a device worn on the head via the one or more sensors in the physical environment.

15 . The method of claim 14 , further comprising:
determining changes in the pose of the head or the device; and updating the positional data corresponding to the 3D positions of the two points on the hand based on detecting rotation or movement of the head or the device based on the determined changes in the pose of the head or the device.

16 . The method of claim 1 , wherein the hand is a first hand, wherein the sensor data is obtained during a period of time and the movement of the hand is determined during the period of time, the method further comprising:
obtaining another sensor data associated with a second hand via the one or more sensors in the physical environment during the period of time; determining additional positional data corresponding to 3D positions of two points on the second hand based on the sensor data during the period of time; and determining the movement of the second hand as the user interaction event based on the additional positional data and whether the movement of the second hand comprises the second hand rotating about the wrist.

17 . The method of claim 1 , wherein determining the movement of the hand as the user interaction event based on the positional data and whether the movement of the hand comprises the hand rotating about the wrist is further based on determining a direction of a gaze.

18 . The method of claim 1 , wherein determining the movement of the hand as the user interaction event based on the positional data and whether the movement of the hand comprises the hand rotating about the wrist is further based on tracking a viewpoint or a pose of the hand.

19 . The method of claim 1 , wherein determining movement of the hand comprises generating a 3D representation of the multiple portions of the hand.

20 . The method of claim 1 , wherein the sensor data comprises multiple sensor data signals.

21 . The method of claim 1 , wherein the sensor data comprises at least one of light intensity image data, depth data, user interface position data, motion data, or a combination thereof.

22 . The method of claim 1 , wherein the device comprises a head mounted device (HMD).

23 . A device comprising:
a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining sensor data associated with a hand via the one or more sensors in a physical environment, the hand comprising a wrist and is associated with an arm; determining positional data corresponding to three-dimensional (3D) positions of two points on the hand based on the sensor data; based on the determined positional data, determining whether a movement of the hand comprises a wrist rotation associated with the hand rotating about the wrist, an arm motion associated with movement of the arm, or a combination of the wrist rotation and the arm motion; and determining the movement of the hand as a user interaction event based on the positional data and whether the movement of the hand comprises the wrist rotation, the arm motion, or a combination of the wrist rotation and the arm motion.

24 . The device of claim 23 , wherein a sensitivity of determining the movement of the hand as the user interaction event is increased based on determining the movement of the hand comprises at least the wrist rotation.

25 . A non-transitory computer-readable storage medium, storing program instructions executable on a device to perform operations comprising:
obtaining sensor data associated with a hand via the one or more sensors in a physical environment, the hand comprising a wrist and is associated with an arm; determining positional data corresponding to three-dimensional (3D) positions of two points on the hand based on the sensor data;

Assignees

• Apple Inc

Inventors

• Brewer Daniel J
• Tang Yirong
• Lortie Chase B

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/017

  Head mounted · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title