Hand gesture-based emojis

Patent metadata

Field	Value
Publication number	US-11340707-B2
Application number	US-202016888562-A
Country	US
Kind code	B2
Filing date	May 29, 2020
Priority date	May 29, 2020
Publication date	May 24, 2022
Grant date	May 24, 2022

Abstract

Examples are disclosed that relate to hand gesture-based emojis. One example provides, on a display device, a method comprising receiving hand tracking data representing a pose of a hand in a coordinate system, based on the hand tracking data, recognizing a hand gesture, and identifying an emoji corresponding to the hand gesture. The method further comprises presenting the emoji on the display device, and sending an instruction to one or more other display devices to present the emoji.

First claim

The invention claimed is:

1. On a display device, a method comprising:
receiving hand tracking data representing a pose of a hand in a coordinate system, the hand tracking data comprising data regarding a spatial position of each of a plurality of joints of a hand model;
based on the hand tracking data, recognizing a hand gesture;
identifying an emoji corresponding to the hand gesture;
presenting the emoji on the display device; and
sending an instruction to one or more other display devices to present the emoji, the instruction instructing the one or more other display devices to present the emoji with a multichannel audio signal that acoustically encodes a spatial origin of the emoji in the coordinate system.

2. The method of claim 1 , wherein the one or more other display devices comprises a targeted display device identified via one or more of gaze tracking and head tracking.

3. The method of claim 1 , wherein the instruction is sent to all display devices sharing the coordinate system.

4. The method of claim 1 , further comprising determining the one or more display devices to which the instruction is sent based upon a setting for the emoji.

5. The method of claim 1 , wherein the instruction specifies an object with which to associate the emoji when presenting the emoji.

6. The method of claim 5 , wherein the object comprises a second display device in the coordinate system, a physical object in the coordinate system, the hand, or a virtual object in the coordinate system.

7. The method of claim 1 , wherein the hand tracking data further indicates a rotation of each of the plurality of joints of the hand model.

8. The method of claim 1 , further comprising disabling recognition of the hand gesture in response to detecting a user interaction with an object.

9. The method of claim 1 , wherein the display device comprises one or more of an augmented reality display and a virtual reality display.

10. The method of claim 1 , further comprising determining the hand tracking data from depth image data.

11. A display device, comprising:
a logic subsystem;
a communication subsystem;
a display subsystem;
a sensor subsystem; and
a storage subsystem comprising instructions executable by the logic subsystem to
receive from the sensor subsystem first articulated hand tracking data representing a pose of a first hand of a first user in a coordinate system, the first articulated hand tracking data comprising data regarding a spatial position of each of a plurality of joints of a hand model;
receive, from a remote device via the communication subsystem, second hand tracking data representing a second hand of a second user in the coordinate system;
compare the first articulated hand tracking data and the second hand tracking data; and
when a condition is met based on the comparing, output an emoji via the display subsystem, the condition comprising a collision between a collider associated with the first hand and a collider associated with the second hand.

12. The display device of claim 11 , wherein the second hand tracking data indicates one or more finger positions and a palm pose.

13. The display device of claim 11 , further comprising instructions executable to display via the display subsystem a representation of the second hand.

14. The display device of claim 11 , further comprising instructions executable to receive eye tracking data indicating a gaze direction of the second user, and to display via the display subsystem a representation of the second user, the representation being animated based on the eye tracking data.

15. The display device of claim 11 , further comprising instructions executable to output a ray representing a pointing direction of the second hand.

16. The display device of claim 11 , further comprising instructions executable to send an instruction to the remote device to present the emoji.

17. The display device of claim 16 , wherein the instruction instructs the remote device to present the emoji with a multichannel audio signal that acoustically encodes a spatial origin of the emoji in the coordinate system.

18. A display device, comprising:
a logic subsystem;
a communication subsystem;
a display subsystem;
a sensor subsystem; and
a storage subsystem comprising instructions executable by the logic subsystem to
receive from the sensor subsystem first articulated hand tracking data representing a pose of a first hand of a first user in a coordinate system, the first articulated hand tracking data comprising data regarding a spatial position of each of a plurality of joints of a hand model;
receive, from a remote device via the communication subsystem, second hand tracking data representing a second hand of a second user in the coordinate system;
compare the first articulated hand tracking data and the second hand tracking data;
when a condition is met based on the comparing, output an emoji via the display subsystem; and
send an instruction to the remote device to present the emoji, the instruction instructing the remote device to present the emoji with a multichannel audio signal that acoustically encodes a spatial origin of the emoji in the coordinate system.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Schwarz Julia
• Notter Michael Harley
• Kam Jenny
• Tang sheng kai
• Jakubzak Kenneth Mitchell
• Behringer Adam Edwin
• Hong Amy Mun
• Neff Joshua Kyle
• Stellmach Sophie
• Lamb Mathew J
• Kamuda Nicholas Ferianc

Classifications

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06V40/107

  Static hand or arm · CPC title