Generating a 3D model of a fingertip for visual touch detection

Patent metadata

Field	Value
Publication number	US-11620790-B2
Application number	US-202117205830-A
Country	US
Kind code	B2
Filing date	Mar 18, 2021
Priority date	Sep 28, 2018
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

Generating a 3D model may include determining, based on sensor data from a touch sensor on a first device, a touch event, wherein the touch event comprises a touch on the first device by a touching object. Generating a 3D model may also include, in response to a touch event, obtaining a first image of the touching object by a first camera of the first device, and obtaining, from a second device, a second image of the touching object, wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object. A model of the touching object is generated based on the first image and the second image.

First claim

The invention claimed is:

1. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain a model of a touching object;
detect, based on device sensor data of a first device, a touch event during a first time, wherein the touch event comprises a touch between the touching object and a target surface;
in response to the touch event:
obtain a first image of the touching object by a first camera of the first device at the first time;
obtain, from a second camera, a second image of the touching object, wherein the second image is captured at the first time;
update the model of the touching object based on at least one of the first image and the second image in accordance with the touch event; and
provide the updated model to a second device, wherein the second camera is comprised in the second device.

2. The non-transitory computer readable medium of claim 1 ,
wherein the updated model is usable by the second device for performing hand tracking.

3. The non-transitory computer readable medium of claim 1 , further comprising computer readable code to:
capture first depth information for the touching object from a depth sensor of the first device,
wherein the computer readable code to update the model of the touching object further comprising computer readable code to update the model of the touching object based on the first depth information.

4. The non-transitory computer readable medium of claim 3 , further comprising computer readable code to, in response to the touch event:
request second depth information for the touching object from a second device, wherein the second camera is comprised in the second device, and
wherein the computer readable code to update the model of the touching object further comprises computer readable code to update the model of the touching object based on the second depth information.

5. The non-transitory computer readable medium of claim 4 , wherein at least one of the first depth information and second depth information is obtained from a perspective at which a touch point on a target surface is occluded by the touching object.

6. The non-transitory computer readable medium of claim 1 , wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object.

7. The non-transitory computer readable medium of claim 1 , wherein the touching object comprises a hand of a user.

8. A method comprising:
obtaining a model of a touching object;
detecting, based on device sensor data of a first device, a touch event during a first time, wherein the touch event comprises a touch between the touching object and a target surface;
in response to the touch event:
obtaining a first image of the touching object by a first camera of the first device at the first time;
obtaining, from a second camera, a second image of the touching object, wherein the second image is captured at the first time;
updating the model of the touching object based on at least one of the first image and the second image in accordance with the touch event; and
providing the updated model to a second device, wherein the second camera is comprised in the second device.

9. The method of claim 8 ,
wherein the updated model is usable by the second device for performing hand tracking.

10. The method of claim 8 , further comprising:
capturing first depth information for the touching object from a depth sensor of the first device,
wherein updating the model of the touching object further comprises updating the model of the touching object based on the first depth information.

11. The method of claim 10 , further comprising, in response to the touch event:
requesting second depth information for the touching object from a second device, wherein the second camera is comprised in the second device, and
wherein updating the model of the touching object further comprises updating the model of the touching object based on the second depth information.

12. The method of claim 11 , wherein at least one of the first depth information and second depth information is obtained from a perspective at which a touch point on a target surface is occluded by the touching object.

13. The method of claim 8 , wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object.

14. The method of claim 8 , wherein the touching object comprises a hand of a user.

15. A system comprising:
one or more processors; and
one or more non-transitory computer readable media comprising computer readable code executable by the one or more processors to:
obtain a model of a touching object;
detect, based on device sensor data of a first device, a touch event during a first time, wherein the touch event comprises a touch between the touching object and a target surface;
in response to the touch event:
obtain a first image of the touching object by a first camera of the first device at the first time;
obtain, from a second camera, a second image of the touching object, wherein the second image is captured at the first time;
update the model of the touching object based on at least one of the first image and the second image in accordance with the touch event; and
provide the updated model to a second device, wherein the second camera is comprised in the second device.

16. The system of claim 15 ,
wherein the updated model is usable by the second device for performing hand tracking.

17. The system of claim 15 , further comprising computer readable code to:
capture first depth information for the touching object from a depth sensor of the first device,
wherein the computer readable code to update the model of the touching object further comprising computer readable code to update the model of the touching object based on the first depth information.

18. The system of claim 17 , further comprising computer readable code to, in response to the touch event:
request second depth information for the touching object from a second device, wherein the second camera is comprised in the second device, and
wherein the computer readable code to update the model of the touching object further comprises computer readable code to update the model of the touching object based on the second depth information.

19. The system of claim 18 , wherein at least one of the first depth information and second depth information is obtained from a perspective at which a touch point on a target surface is occluded by the touching object.

20. The system of claim 15 , wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object.

Assignees

• Apple Inc

Inventors

• SETHI Rohit
• Wang Lejing

Classifications

• G06T15/205

  Image-based rendering · CPC title

• G06V2201/12

  Acquisition of 3D measurements of objects · CPC title

• G06F3/0488 Primary

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G06T1/0007

  Image acquisition · CPC title

• G06F3/041

  Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means · CPC title