Pinch recognition using finger zones

Patent metadata

Field	Value
Publication number	US-12229344-B2
Application number	US-202318478183-A
Country	US
Kind code	B2
Filing date	Sep 29, 2023
Priority date	Jun 2, 2023
Publication date	Feb 18, 2025
Grant date	Feb 18, 2025

Abstract

Enabling gesture input includes obtaining hand tracking data based on one or more camera frames, detecting a contact event between a first finger and a second finger based on the hand tracking data, and determining a first contact location on the first finger and a second contact location on the second finger. In accordance with a determination that the first contact location and the second contact location are within a first predefined gesture zone for a first gesture, an input action is enabled corresponding to the first gesture.

First claim

The invention claimed is:

1. A method comprising:
obtaining hand tracking data based on a series of camera frames;
detecting a touch between a first finger and a second finger in a first frame based on the hand tracking data;
detecting a first contact location for the touch at a first point on the first finger;
in accordance with a determination that the first contact location is within a predefined gesture zone for a first gesture, initiating a gesture-based input event corresponding to the first gesture;
detecting the touch in a second frame at a second contact location at a second point on the first finger; and
cancelling the gesture-based input event in accordance with a difference between the first point and the second point on the first finger satisfying a cancellation criterion.

2. The method of claim 1 , wherein a contact event comprises the touch and a release of the touch, and wherein the gesture-based input event comprises a visual feedback and an input action.

3. The method of claim 2 , wherein the first contact location is within a first predefined gesture zone for the first gesture in the first frame, and
wherein the input action is cancelled in accordance with the second contact location being outside the first predefined gesture zone for the first gesture in the second frame.

4. The method of claim 2 ,
wherein cancelling the gesture-based input event in accordance with a difference comprises determining that a distance between the first contact location and the second contact location satisfies a predetermined threshold distance within a predefined time period.

5. The method of claim 1 , wherein the first gesture comprises a pinch.

6. The method of claim 1 , wherein obtaining the hand tracking data comprises obtaining a series of camera frames, wherein the touch is detected based on the series of camera frames, and wherein a contact determination is performed for each frame of the series of camera frames.

7. The method of claim 2 , wherein the touch triggers the visual feedback of the gesture-based input event.

8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain hand tracking data based on one or more camera frames;
detect a touch between a first finger and a second finger in a first frame based on the hand tracking data;
detect a first contact location for the touch at a first point on the first finger;
in accordance with a determination that the first contact location is within a first predefined gesture zone for a first gesture, initiate a gesture-based input event corresponding to the first gesture;
detect the touch in a second frame at a second contact location at a second point on the first finger; and
cancel the gesture-based input event in accordance with a difference between the first point and the second point on the first finger satisfying a cancellation criterion.

9. The non-transitory computer readable medium of claim 8 , wherein a contact event comprises the touch and a release of the touch, and wherein the gesture-based input event comprises a visual feedback and an input action.

10. The non-transitory computer readable medium of claim 9 ,
wherein the first contact location is within a first predefined gesture zone for the first gesture in the first frame, and
wherein the input action is cancelled in accordance with the second contact location being outside the first predefined gesture zone for the first gesture in the second frame.

11. The non-transitory computer readable medium of claim 9 , wherein the computer readable code to cancel the gesture-based input event in accordance with a difference comprises computer readable code to determine that a distance between the first contact location and the second contact location satisfies a predetermined threshold distance within a predefined time period.

12. The non-transitory computer readable medium of claim 8 , wherein the first gesture comprises a pinch.

13. The non-transitory computer readable medium of claim 8 , wherein the computer readable code to obtain the hand tracking data comprises computer readable code to obtain a series of camera frames, wherein the touch is detected based on the series of camera frames, and wherein a contact determination is performed for each frame of the series of camera frames.

14. The non-transitory computer readable medium of claim 9 , wherein the touch triggers the visual feedback of the gesture-based input event.

15. A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain hand tracking data based on one or more camera frames;
detect a touch between a first finger and a second finger in a first frame based on the hand tracking data;
detect a first contact location for the touch at a first point the first finger;
in accordance with a determination that the first contact location is within a first predefined gesture zone for a first gesture, initiate a gesture-based input event corresponding to the first gesture;
detect the touch in a second frame at a second contact location at a second point on the first finger; and

cancel the gesture-based input event in accordance with a difference between the first point and the second point on the first finger satisfying a cancellation criterion.

16. The system of claim 15 , wherein a contact event comprises the touch and a release of the touch, and wherein the gesture-based input event comprises a visual feedback and an input action.

17. The system of claim 16 ,
wherein the first contact location is within a predefined gesture zone for the first gesture in the first frame, and
wherein the input action is cancelled in accordance with the second contact location being outside the first predefined gesture zone for the first gesture in the second frame.

18. The system of claim 16 , code to:
wherein the computer readable code to cancel the gesture-based input event in accordance with a difference comprises computer readable code to determine that a distance between the first contact location and the second contact location satisfies a predetermined threshold distance within a predefined time period.

19. The system of claim 15 , wherein the computer readable code to obtain the hand tracking data comprises computer readable code to obtain a series of camera frames, wherein the touch is detected based on the series of camera frames, and wherein a contact determination is performed for each frame of the series of camera frames.

20. The system of claim 16 , wherein the touch triggers the visual feedback of the gesture-based input event.

Assignees

• Apple Inc

Inventors

• Belyaev Victor
• Nayak Bhavin Vinodkumar
• Brewer Daniel J
• Bar Yosef Itay
• Shutzberg Julian K
• Schroeder Matthias M

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T7/248

  involving reference images or patches · CPC title

• G06T7/215

  Motion-based segmentation · CPC title