Generating a 3D model of a fingertip for visual touch detection

Patent metadata

Field	Value
Publication number	US-10970922-B2
Application number	US-201916580873-A
Country	US
Kind code	B2
Filing date	Sep 24, 2019
Priority date	Sep 28, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Generating a 3D model may include determining, based on sensor data from a touch sensor on a first device, a touch event, wherein the touch event comprises a touch on the first device by a touching object. Generating a 3D model may also include, in response to a touch event, obtaining a first image of the touching object by a first camera of the first device, and obtaining, from a second device, a second image of the touching object, wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object. A model of the touching object is generated based on the first image and the second image.

First claim

The invention claimed is:

1. A computer readable medium comprising computer readable code executable by one or more processors to:
determine, based on sensor data from a touch sensor on a first device, a touch event, wherein the touch event comprises a touch on the first device by a touching object, and wherein the touch event occurs during a first time;
in response to a touch event:
obtain a first image of the touching object by a first camera of the first device during the first time, and
obtain, from a second device, a second image of the touching object captured during the first time,
wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object; and

cause a model of the touching object to be generated based on the first image and the second image.

2. The computer readable medium of claim 1 , further comprising computer readable code to, in response to the touch event:
capture first depth information for the touching object from a depth sensor of the first device,
wherein the computer readable code to generate the model of the touching object further comprising computer readable code to generate the model of the touching object based on the first depth information.

3. The computer readable medium of claim 1 , further comprising computer readable code to, in response to the touch event:
request second depth information for the touching object from the second device,
wherein the computer readable code to generate the model of the touching object further comprises computer readable code to generate the model of the touching object based on the second depth information.

4. The computer readable medium of claim 1 , wherein the computer readable code to generate the model of the touching object comprises computer readable code to:
transmit the first image and the second image to a base station; and
cause the base station to generate the model of the touching object.

5. The computer readable medium of claim 1 , further comprising:
providing the model of the touching object to the second device.

6. The computer readable medium of claim 1 , wherein the touching object is an appendage of a user of the first device and the second device.

7. The computer readable medium of claim 1 , further comprising computer readable code to:
detect, by the touch sensor on the first device, an additional touch event, wherein the additional touch event comprises a second touch on the first device by the touching object;
in response to the additional touch event:
obtain a third image of the touching object by the first camera of the first device;
request, from the second device, a fourth image of the touching object,
wherein the third image of the touching object captures a third view of the touching object, and wherein the fourth image of the touching object captures a fourth view of the touching object; and

cause the model to be updated based on the third image and the fourth image.

8. A system for generating a 3D model, comprising:
one or more processors; and
a memory coupled to the one or more processors and comprising computer readable medium comprising computer readable code executable by the one or more processors to:
determine, based on sensor data from a touch sensor on a first device, a touch event, wherein the touch event comprises a touch on the first device by a touching object, and wherein the touch event occurs during a first time;
in response to a touch event:
obtain, during the first time, a first image of the touching object by a first camera of the first device, and
obtain, from a second device, a second image of the touching object captured during the first time,
wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object; and

cause a model of the touching object to be generated based on the first image and the second image.

9. The system of claim 8 , further comprising computer readable code to, in response to the touch event:
capture first depth information for the touching object from a depth sensor of the first device,
wherein the computer readable code to generate the model of the touching object further comprising computer readable code to generate the model of the touching object based on the first depth information.

10. The system of claim 8 , further comprising computer readable code to, in response to the touch event:
request second depth information for the touching object from the second device,
wherein the computer readable code to generate the model of the touching object further comprises computer readable code to generate the model of the touching object based on the second depth information.

11. The system of claim 8 , wherein the computer readable code to generate the model of the touching object comprises computer readable code to:
transmit the first image and the second image to a base station; and
cause the base station to generate the model of the touching object.

12. The system of claim 8 , further comprising:
providing the model of the touching object to the second device.

13. The system of claim 8 , wherein the touching object is an appendage of a user of the first device and the second device.

14. The system of claim 8 , further comprising computer readable code to:
detect, by the touch sensor on the first device, an additional touch event, wherein the additional touch event comprises a second touch on the first device by the touching object;
in response to the additional touch event:
obtain a third image of the touching object by the first camera of the first device;
request, from the second device, a fourth image of the touching object,
wherein the third image of the touching object captures a third view of the touching object, and wherein the fourth image of the touching object captures a fourth view of the touching object; and

cause the model to be updated based on the third image and the fourth image.

15. A method for generating a 3D model, comprising:
determining, based on sensor data from a touch sensor on a first device, a touch event, wherein the touch event comprises a touch on the first device by a touching object, and wherein the touch event occurs during a first time;
in response to a touch event:
obtaining, during the first time, a first image of the touching object by a first camera of the first device, and
obtaining, from a second device, a second image of the touching object captured during the first time,
wherein the first image of the touching object captures a first view of the touching object, and wherein the second image of the touching object captures a second view of the touching object; and

causing a model of the touching object to be generated based on the first image and the second image.

16. The method of claim 15 , further comprising, in response to the touch event:
capturing first depth information for the touching object from a depth sensor of the first device,
wherein generating the model of the touching object further comprises generating the model of the touching object based on the first depth information.

17. The method of claim 15 , further comprising, in response to the touch event:
requesting second depth information for the touching object from the second device,
wherein generating the model of the touching object further comprises generating the model of the touching object

Assignees

• Apple Inc

Inventors

• SETHI Rohit
• Wang Lejing

Classifications

• G06V2201/12

  Acquisition of 3D measurements of objects · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/041

  Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means · CPC title