Heuristic for hand tracking correction modes

Patent metadata

Field	Value
Publication number	US-12265665-B1
Application number	US-202318371098-A
Country	US
Kind code	B1
Filing date	Sep 21, 2023
Priority date	Sep 23, 2022
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

In some implementations, a method includes: determining a location for virtual content; detecting a user interaction with the virtual content; in response to detecting the user interaction with the virtual content, determining a position of a hand gesture during the user interaction with the virtual content; in accordance with a determination that the position of the hand gesture is within a threshold distance relative to the location of the virtual content, generating corrected hand tracking data associated with the user interaction with the virtual content; and in accordance with a determination that the position of the hand gesture is outside of the threshold distance relative to the location of the virtual content, generating uncorrected hand tracking data associated with the user interaction with the virtual content.

First claim

What is claimed is:

1. A method comprising:
at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices via a communication interface:
determining a location for virtual content;
detecting a user interaction with the virtual content;
in response to detecting the user interaction with the virtual content, determining a position of a hand gesture during the user interaction with the virtual content;
in accordance with a determination that the position of the hand gesture is within a threshold distance relative to the location of the virtual content, generating corrected hand tracking data associated with the user interaction with the virtual content and corresponding to a direct interaction with the virtual content; and
in accordance with a determination that the position of the hand gesture is outside of the threshold distance relative to the location of the virtual content, generating uncorrected hand tracking data associated with the user interaction with the virtual content and corresponding to an indirect interaction with the virtual content.

2. The method of claim 1 , wherein generating corrected hand tracking data associated with the user interaction with the virtual content includes:
obtaining uncorrected hand tracking data associated with a current time period and uncorrected hand tracking data associated with a previous time period;
obtaining corrected hand tracking data associated with the current time period;
generating differential hand tracking data by performing a difference between the corrected hand tracking data associated with the current time period and the uncorrected hand tracking data associated with the previous time period;
generating temporally smoothed differential hand tracking data by performing temporal smoothing on the differential hand tracking data based on a depth map of a physical environment for the current time period; and
generating output hand tracking data by performing a difference between the temporally smoothed differential hand tracking data and the uncorrected hand tracking data associated with the current time period, wherein the output hand tracking data corresponds to the corrected hand tracking data associated with the user interaction with the virtual content.

3. The method of claim 2 , wherein obtaining the corrected hand tracking data associated with the current time period includes:
obtaining a first set of two-dimensional coordinates of a hand of the user in a physical environment; and
transforming the first set of two-dimensional coordinates into a second set of two-dimensional coordinates based on a depth map of the physical environment.

4. The method of claim 3 , wherein the first set of two-dimensional coordinates are generated by projecting first three-dimensional coordinates onto an image plane.

5. The method of claim 4 , wherein the first set of two-dimensional coordinates includes a left set and a right set used for triangulation.

6. The method of claim 4 , wherein a second set of three-dimensional coordinates is generated from the second set of two-dimensional coordinates.

7. The method of claim 2 , further comprising:
rendering the user interaction with the virtual content based on the output hand tracking data; and
displaying, via the display device, the rendered user interaction with the virtual content based on the output hand tracking data.

8. The method of claim 1 , wherein, in accordance with a determination that the position of the hand gesture is within a threshold distance relative to the location of the virtual content, the user interaction with the virtual content corresponds to a direct interaction type and the user interaction corresponds to a hand tracking input relative to the virtual object.

9. The method of claim 1 , wherein, in accordance with a determination that the position of the hand gesture is outside of the threshold distance relative to the location of the virtual content, the user interaction with the virtual content corresponds to an indirect interaction type and the user interaction corresponds to one of an eye tracking input, a voice input, or a gestural input relative to the virtual object.

10. A computing system comprising:
one or more processors;
a non-transitory memory;
an interface for communicating with a display device and one or more input devices; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the computing system to:
determine a location for virtual content;
detect a user interaction with the virtual content;
in response to detecting the user interaction with the virtual content, determine a position of a hand gesture during the user interaction with the virtual content;
in accordance with a determination that the position of the hand gesture is within a threshold distance relative to the location of the virtual content, generate corrected hand tracking data associated with the user interaction with the virtual content and corresponding to a direct interaction with the virtual content; and
in accordance with a determination that the position of the hand gesture is outside of the threshold distance relative to the location of the virtual content, generate uncorrected hand tracking data associated with the user interaction with the virtual content and corresponding to an indirect interaction with the virtual content.

11. The computing system of claim 10 , wherein generating corrected hand tracking data associated with the user interaction with the virtual content includes:
obtaining uncorrected hand tracking data associated with a current time period and uncorrected hand tracking data associated with a previous time period;
obtaining corrected hand tracking data associated with the current time period;
generating differential hand tracking data by performing a difference between the corrected hand tracking data associated with the current time period and the uncorrected hand tracking data associated with the previous time period;
generating temporally smoothed differential hand tracking data by performing temporal smoothing on the differential hand tracking data based on a depth map of a physical environment for the current time period; and
generating output hand tracking data by performing a difference between the temporally smoothed differential hand tracking data and the uncorrected hand tracking data associated with the current time period, wherein the output hand tracking data corresponds to the corrected hand tracking data associated with the user interaction with the virtual content.

12. The computing system of claim 11 , wherein obtaining the corrected hand tracking data associated with the current time period includes:
obtaining a first set of two-dimensional coordinates of a hand of the user in a physical environment; and
transforming the first set of two-dimensional coordinates into a second set of two-dimensional coordinates based on a depth map of the physical environment.

13. The computing system of claim 10 , wherein, in accordance with a determination that the position of the hand gesture is within a threshold distance relative to the location of the virtual content, the user interaction with the virtual content corresponds to a direct interaction type and the user interaction corresponds to a hand tracking input relative to the virtual object.

14. The computing system of claim 10 , wherein, in accordance with a determination that the position of the hand gesture is outside of the thresh

Assignees

• Apple Inc

Inventors

• Piuze-Phaneuf Emmanuel
• Ercan Ali
• Shutzberg Julian K
• Lacey Paul A

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title