Finger orientation touch detection

Patent metadata

Field	Value
Publication number	US-11934584-B2
Application number	US-202017022583-A
Country	US
Kind code	B2
Filing date	Sep 16, 2020
Priority date	Sep 27, 2019
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

Touch detection includes obtaining image data for a finger and a target surface, and determining a touch region in the image data on which the fingertip makes contact with the target surface. A pointing direction of the finger in the image data is determined, and a target region is estimated on the target surface based on the touch region and the pointing direction, wherein the touch region comprises a portion of the target surface that extends from the touch region in a direction of the pointing direction.

First claim

The invention claimed is:

1. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain image data for a finger and a target surface;
identify a touch region on the target surface where the finger makes contact with the target surface;
determine a directional 3D ray consistent with a pointing direction of the finger in the image data;
estimate a target region based on a point in space at which the directional 3D ray passes through the target surface, wherein the point in space is determined based on an estimated depth of the target surface, and wherein the target region is different from the touch region;
determine a touch status based on the estimated depth of the target surface; and
trigger an event based on the touch status, wherein the event is based on the target region.

2. The non-transitory computer readable medium of claim 1 , further comprising computer readable code to:
estimate a geometric property of the target surface based on the estimated target region;
process surface depth information of the target region based on the geometric property; and
determine the touch status based on the target surface depth information.

3. The non-transitory computer readable medium of claim 2 , further comprising computer readable code to:
provide digital information associated with the target region; and
trigger an event based on the associated digital information and the touch status.

4. The non-transitory computer readable medium of claim 1 , wherein the computer readable code to determine the pointing direction of the finger further comprises computer readable code to:
detect a first joint and a second joint of the finger in the image data,
wherein the pointing direction is further determined according to a point in space of each of the first joint and the second joint.

5. The non-transitory computer readable medium of claim 1 , wherein the computer readable code to determine the pointing direction of the finger further comprises computer readable code to:
detect a hand position in the image data,
wherein the pointing direction is further determined according to the hand position.

6. The non-transitory computer readable medium of claim 1 , further comprises computer readable code to:
detect a gaze vector of a user associated with the finger; and
refine the target region according to the gaze vector.

7. The non-transitory computer readable medium of claim 1 , wherein the image data comprises a depth image obtained by a head mounted device.

8. A system for detecting touch, comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
obtain image data for a finger and a target surface;
identify a touch region on the target surface where the finger makes contact with the target surface;
determine a directional 3D ray consistent with a pointing direction of the finger in the image data;
estimate a target region based on a point in space at which the directional 3 D ray passes through the target surface, wherein the point in space is determined based on an estimated depth of the target surface, and wherein the target region is different from the touch region;
determine a touch status based on the estimated depth information; and
trigger an event based on the touch status, wherein the event is determined based on the target region.

9. The system of claim 8 , further comprising computer readable code to:
estimate a geometric property of the target surface based on the estimated target region;
process surface depth information of the target region based on the geometric property; and
determine the touch status based on the target surface depth information.

10. The system of claim 9 , further comprising computer readable code to:
provide digital information associated with the target region; and
trigger an event based on the associated digital information and the touch status.

11. The system of claim 8 , wherein the computer readable code to determine the pointing direction of the finger further comprises computer readable code to:
detect a first joint and a second joint of the finger in the image data,
wherein the pointing direction is further determined according to a point in space of each of the first joint and the second joint.

12. The system of claim 8 , wherein the computer readable code to determine the pointing direction of the finger further comprises computer readable code to:
detect a hand position in the image data,
wherein the pointing direction is further determined according to the hand position.

13. The system of claim 8 , further comprises computer readable code to:
detect a gaze vector of a user associated with the finger; and
refine the target region according to the gaze vector.

14. A method for detecting touch, comprising:
obtaining image data for a finger and a target surface;
identifying a touch region on the target surface where the finger makes contact with the target surface;
determining a directional 3D ray consistent with a pointing direction of the finger in the image data;
estimating a target region based on a point in space at which the directional 3D ray passes through the target surface, wherein the point in space is determined based on an estimated depth of the target surface, and wherein the target region is different from the touch region;
determining a touch status based on the estimated depth of the target surface; and
triggering an event based on the touch status, wherein the event is based on the target region.

15. The method of claim 14 , further comprising:
estimating a geometric property of the target surface based on the estimated target region;
processing surface depth information of the target region based on the geometric property; and
determining the touch status based on the target surface depth information.

16. The method of claim 15 , further comprising:
providing digital information associated with the target region; and
triggering an event based on the associated digital information and the touch status.

17. The method of claim 14 , wherein determining the pointing direction of the finger further comprises:
detecting a first joint and a second joint of the finger in the image data,
wherein the pointing direction is further determined according to a point in space of each of the first joint and the second joint.

18. The method of claim 14 , wherein determining the pointing direction of the finger further comprises:
detecting a hand position in the image data,
wherein the pointing direction is further determined according to the hand position.

19. The method of claim 14 , wherein determining the pointing direction of the finger further comprises:
detecting a gaze vector of a user associated with the finger; and
refining the target region according to the gaze vector.

20. The method of claim 14 , wherein the image data comprises a depth image obtained by a head mounted device.

Assignees

• Apple Inc

Inventors

• Wang Lejing
• Meloun Martin
• Lim Ji Hyoun

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title