IMU for touch detection

Patent metadata

Field	Value
Publication number	US-11803233-B2
Application number	US-202217663690-A
Country	US
Kind code	B2
Filing date	May 17, 2022
Priority date	Sep 28, 2018
Publication date	Oct 31, 2023
Grant date	Oct 31, 2023

Abstract

Touch detection may include determining, based on data from an IMU on a first device that monitors movement of a touching object, a touch event, wherein the touch event indicates contact between the touching object and a surface, obtaining a depth image captured by a second device, comprising a touch image, determining a touch point of the object based on the touch image, and providing a response based on the touch point of the object and the touched surface.

First claim

The invention claimed is:

1. A method comprising:
receiving, at a first device, from an inertial measurement unit (IMU) on a second device that monitors movement of a touching object, an indication of a potential touch event;
obtaining, by the first device and in response to receiving the indication, an image of a scene comprising the touching object, wherein the image is captured coincident with the touch event;
determining, based on the image, whether the touching object performed a touch event; and

generating a response based on the determination.

2. The method of claim 1 , wherein, in accordance with a determination touch event occurred between the touching object and a surface, generating the response comprises:
generating virtual information in accordance with the touch event, and
presenting the virtual information on a display.

3. The method of claim 1 , wherein, in accordance with a determination touch event did not occur, generating the response comprises:
presenting an indication of a false positive in accordance with the potential touch event.

4. The method of claim 1 , wherein the indication comprises a time stamp indicating a time of the potential touch event, and wherein obtaining the touch image comprises:
retrieving the touch image from cached depth data in accordance with the time stamp.

5. The method of claim 1 , wherein the response is generated based on a touched portion of a surface.

6. The method of claim 1 , wherein generating the response comprises:
identifying, from the image, a predetermined gesture,
wherein the response is generated based on the identified gesture.

7. The method of claim 1 , wherein the second device is a wearable device.

8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
receive, at a first device, from an inertial measurement unit (IMU) on a second device that monitors movement of a touching object, an indication of a potential touch event;
obtain, by the first device and in response to receiving the indication, an image of a scene comprising the touching object, wherein the image is captured coincident with the touch event;
determine, based on the image, whether the touching object performed a touch event; and
generate a response based on the determination.

9. The non-transitory computer readable medium of claim 8 , wherein, in accordance with a determination touch event occurred between the touching object and a surface, the computer readable code to generate the response comprises computer readable code to:
generate virtual information in accordance with the touch event, and
present the virtual information on a display.

10. The non-transitory computer readable medium of claim 8 , wherein, in accordance with a determination touch event did not occur, the computer readable code to generate the response comprises computer readable code to:
present an indication of a false positive in accordance with the potential touch event.

11. The non-transitory computer readable medium of claim 8 , wherein the indication comprises a time stamp indicating a time of the potential touch event, and wherein the computer readable code to obtain the image comprises computer readable code to:
retrieve the image from cached depth data in accordance with the time stamp.

12. The non-transitory computer readable medium of claim 8 , wherein the response is generated based on a touched portion of a surface.

13. The non-transitory computer readable storage medium of claim 8 , wherein the computer readable code to generate the response comprises computer readable code to:
identify, from the image, a predetermined gesture,
wherein the response is generated based on the identified gesture.

14. The non-transitory computer readable medium of claim 8 , wherein the second device is a wearable device.

15. A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by one or more processors to:
receive, at a first device, from an inertial measurement unit (IMU) on a second device that monitors movement of a touching object, an indication of a potential touch event;
obtain, by the first device and in response to receiving the indication, an image of a scene comprising the touching object, wherein the image is captured coincident with the touch event;
determine, based on the image, whether the touching object performed a touch event; and
generate a response based on the determination.

16. The system of claim 15 , wherein, in accordance with a determination touch event occurred between the touching object and the surface, the computer readable code to generate the response comprises computer readable code to:
generate virtual information in accordance with the touch event, and
present the virtual information on a display.

17. The system of claim 15 , wherein, in accordance with a determination touch event did not occur, the computer readable code to generate the response comprises computer readable code to:
present an indication of a false positive in accordance with the potential touch event.

18. The system of claim 15 , wherein the indication comprises a time stamp indicating a time of the potential touch event, and wherein the computer readable code to obtain the image comprises computer readable code to:
retrieve the image from cached depth data in accordance with the time stamp.

19. The system of claim 15 , wherein the response is generated based on a touched portion of a surface.

20. The system of claim 15 , wherein the computer readable code to generate the response comprises computer readable code to:
identify, from the image, a predetermined gesture,
wherein the response is generated based on the identified gesture.

Assignees

• Apple Inc

Inventors

• Wang Lejing
• SETHI Rohit

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title

• G06T1/0007

  Image acquisition · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title