Spatiotemporal Smoothing for Improved Hand Tracking

Patent metadata

Field	Value
Publication number	US-2025155988-A1
Application number	US-202519022868-A
Country	US
Kind code	A1
Filing date	Jan 15, 2025
Priority date	Sep 23, 2022
Publication date	May 15, 2025
Grant date	—

Abstract

In some implementations, a method includes: obtaining uncorrected hand tracking data; obtaining a depth map associated with a physical environment; identifying a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; performing spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generating corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger.

First claim

What is claimed is:

1 . A method comprising:
at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices via a communication interface:
obtaining uncorrected hand tracking data;
obtaining a depth map associated with a physical environment;
identifying a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data;
performing spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and
generating corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger.

2 . The method of claim 1 , wherein the portion of the finger corresponds to one of a fingertip, a particular knuckle, or a centroid of a finger.

3 . The method of claim 1 , wherein the position of the portion of the finger is identified by projecting the portion of a finger into a depth space associated with the depth map.

4 . The method of claim 1 , wherein the region of the depth map adjacent to the portion of the finger corresponds to an N×M pixel area centered on the projection of the portion of the finger.

5 . The method of claim 1 , wherein the region of the depth map to the portion of the finger corresponds to a predefined radius centered on the projection of the portion of the finger.

6 . The method of claim 1 , wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes performing POV correction on each joint within the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger.

7 . The method of claim 1 , wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes:
obtaining a first set of two-dimensional coordinates of a hand of the user in the physical environment; and transforming the first set of two-dimensional coordinates into a second set of two-dimensional coordinates based on the spatially depth smoothed region adjacent to the portion of the finger.

8 . The method of claim 7 , wherein the first set of two-dimensional coordinates are generated by projecting first three-dimensional coordinates onto an image plane.

9 . The method of claim 7 , wherein the first set of two-dimensional coordinates includes a left set and a right set used for triangulation.

10 . The method of claim 7 , further comprising:
presenting, via the display device, hand locations at the second set of two-dimensional coordinates.

11 . The method of claim 7 , wherein a second set of three-dimensional coordinates is generated from the second set of two-dimensional coordinates.

12 . The method of claim 11 , further comprising:
determining a user interaction based on the hand location at the second set of three-dimensional coordinates interacting with a virtual object at the three-dimensional coordinates.

13 . The method of claim 1 , further comprising:
rendering a user interaction with a virtual object based on the corrected hand tracking data; and presenting, via the display device, the rendered user interaction with the virtual object based on the corrected hand tracking data.

14 . A computing system comprising:
one or more processors; a non-transitory memory; an interface for communicating with a display device and one or more input devices; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the computing system to:
obtain uncorrected hand tracking data;
obtain a depth map associated with a physical environment;
identify a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data;
perform spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and
generate corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger.

15 . The computing system of claim 14 , wherein the region of the depth map adjacent to the portion of the finger corresponds to an N×M pixel area centered on the projection of the portion of the finger.

16 . The computing system of claim 14 , wherein the region of the depth map to the portion of the finger corresponds to a predefined radius centered on the projection of the portion of the finger.

17 . The computing system of claim 14 , wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes performing POV correction on each joint within the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger.

18 . The computing system of claim 14 , wherein the one or more programs further cause the computing system to:
render a user interaction with a virtual object based on the corrected hand tracking data; and present, via the display device, the rendered user interaction with the virtual object based on the corrected hand tracking data.

19 . A non-transitory memory storing one or more programs, which, when executed by one or more processors of a computing system with an interface for communicating with a display device and one or more input devices, cause the computing system to:
obtain uncorrected hand tracking data; obtain a depth map associated with a physical environment; identify a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; perform spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generate corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger.

20 . The non-transitory memory of claim 19 , wherein the region of the depth map adjacent to the portion of the finger corresponds to an N×M pixel area centered on the projection of the portion of the finger.

21 . The non-transitory memory of claim 19 , wherein the region of the depth map to the portion of the finger corresponds to a predefined radius centered on the projection of the portion of the finger.

22 . The non-transitory memory of claim 19 , wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes performing POV correction on each joint within the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger.

23 . The non-transitory memory of claim 19 , wherein the one or more programs further cause the computing system to:
render a user interaction with a virtual object based on the corrected hand tracking data; and present, via the display device, the rendered user

Assignees

• Apple Inc

Inventors

• Piuze-Phaneuf Emmanuel
• Ercan Ali
• Shutzberg Julian K
• Lacey Paul A

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/73

  using feature-based methods · CPC title