IMU for touch detection

Patent metadata

Field	Value
Publication number	US-11360550-B2
Application number	US-202117355320-A
Country	US
Kind code	B2
Filing date	Jun 23, 2021
Priority date	Sep 28, 2018
Publication date	Jun 14, 2022
Grant date	Jun 14, 2022

Abstract

Touch detection may include determining, based on data from an IMU on a first device that monitors movement of a touching object, a touch event, wherein the touch event indicates contact between the touching object and a surface, obtaining a depth image captured by a second device, comprising a touch image, determining a touch point of the object based on the touch image, and providing a response based on the touch point of the object and the touched surface.

First claim

The invention claimed is:

1. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
detect, at a first device, an indication that a touch event has occurred based on data received from an inertial measurement unit (IMU) on a second device that monitors movement of a touching object, wherein the touch event indicates contact between the touching object and a surface;
in response to a touch event:
obtain, by the first device, a touch image of the touching object, wherein the touching object is making contact with the surface in the touch image;
detect an indication of a change in direction of the touching object based on additional data received from the IMU;
capture a non-touch image of the touching object, wherein the touching object is not making contact with the surface in the non-touch image; and
generate a model of the touching object based on the touch image and the non-touch image.

2. The non-transitory computer readable medium of claim 1 , wherein the touching object comprises a finger and wherein the touch event occurs between the finger and the surface.

3. The non-transitory computer readable medium of claim 1 , wherein the computer readable code to detect the indication that the touch event has occurred further comprise computer readable code to:
determine that the touch event is a true touch event based on depth information from the touch image for the touching object and the surface.

4. The non-transitory computer readable medium of claim 3 , wherein the computer readable code to capture the non-touch image further comprises computer readable code to:
cause the non-touch image to be captured in accordance with a determination that the touch event is a true touch event, and a determination that the touching object has changed directions.

5. The non-transitory computer readable medium of claim 1 , further comprising computer readable code to:
determine a pose of the touching object at a touch point on the surface, wherein the model of the touching object is further generated based on the pose of the touching object.

6. The non-transitory computer readable medium of claim 5 , further comprising computer readable code to:
provide the model of the touching object to the second device for visual tracking of the touching object.

7. The non-transitory computer readable medium of claim 1 , wherein the second device is a wearable device.

8. A method comprising:
detecting, at a first device, an indication that a touch event has occurred based on data received from an inertial measurement unit (IMU) on a second device that monitors movement of a touching object, wherein the touch event indicates contact between the touching object and a surface;
in response to a touch event:
obtaining, by the first device, a touch image of the touching object, wherein the touching object is making contact with the surface in the touch image;
detecting an indication of a change in direction of the touching object based on additional data received from the IMU;
capturing a non-touch image of the touching object, wherein the touching object is not making contact with the surface in the non-touch image; and
generating a model of the touching object based on the touch image and the non-touch image.

9. The method of claim 8 , wherein the touching object comprises a finger and wherein the touch event occurs between the finger and the surface.

10. The method of claim 8 , wherein detecting the indication that the touch event has occurred further comprises:
determining that the touch event is a true touch event based on depth information from the touch image for the touching object and the surface.

11. The method of claim 10 , wherein capturing the non-touch image further comprises:
causing the non-touch image to be captured in accordance with a determination that the touch event is a true touch event, and a determination that the touching object has changed directions.

12. The method of claim 8 , further comprising:
determining a pose of the touching object at a touch point on the surface, wherein the model of the touching object is further generated based on the pose of the touching object.

13. The method of claim 12 , further comprising:
providing the model of the touching object to the second device for visual tracking of the touching object.

14. The method of claim 8 , wherein the second device is a wearable device.

15. A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
detect, at a first device, an indication that a touch event has occurred based on data received from an inertial measurement unit (IMU) on a second device that monitors movement of a touching object, wherein the touch event indicates contact between the touching object and a surface;
in response to a touch event:
obtain, by the first device, a touch image of the touching object, wherein the touching object is making contact with the surface in the touch image;
detect an indication of a change in direction of the touching object based on additional data received from the IMU;
capture a non-touch image of the touching object, wherein the touching object is not making contact with the surface in the non-touch image; and
generate a model of the touching object based on the touch image and the non-touch image.

16. The system of claim 15 , wherein the touching object comprises a finger and wherein the touch event occurs between the finger and the surface.

17. The system of claim 15 , wherein the computer readable code to detect the indication that the touch event has occurred further comprise computer readable code to:
determine that the touch event is a true touch event based on depth information from the touch image for the touching object and the surface.

18. The system of claim 17 , wherein the computer readable code to capture the non-touch image further comprises computer readable code to:
cause the non-touch image to be captured in accordance with a determination that the touch event is a true touch event, and a determination that the touching object has changed directions.

19. The system of claim 15 , further comprising computer readable code to:
determine a pose of the touching object at a touch point on the surface, wherein the model of the touching object is further generated based on the pose of the touching object.

20. The system of claim 19 , further comprising computer readable code to:
provide the model of the touching object to the second device for visual tracking of the touching object.

Assignees

• Apple Inc

Inventors

• Wang Lejing
• SETHI Rohit

Classifications

• G06F3/0416 Primary

  Control or interface arrangements specially adapted for digitisers · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G06T2207/30196

  Human being; Person · CPC title