Eyewear including sign language to speech translation

Patent metadata

Field	Value
Publication number	US-11900729-B2
Application number	US-202117530175-A
Country	US
Kind code	B2
Filing date	Nov 18, 2021
Priority date	Dec 16, 2020
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

An eyewear having an electronic processor configured to identify a hand gesture including a sign language, and to generate speech that is indicative of the identified hand gesture. The electronic processor uses a convolutional neural network (CNN) to identify the hand gesture by matching the hand gesture in the image to a set of hand gestures, wherein the set of hand gestures is a library of hand gestures stored in a memory. The hand gesture can include a static hand gesture, and a moving hand gesture. The electronic processor is configured to identify a word from a series of hand gestures.

First claim

What is claimed is:

1. Eyewear, comprising:
a frame configured to be worn on a head of a user;
a speaker supported by the frame;
a camera supported by the frame and configured to generate an image including a hand gesture of a user of the eyewear; and
an electronic processor configured to:
receive the image including the hand gesture from the camera;
identify the hand gesture as a sign language; and
generate speech via the speaker that is indicative of the identified hand gesture of the user, wherein the speech is configured to be heard and understood by a person to enable a conversation between the user and the person.

2. The eyewear of claim 1 , wherein the electronic processor is configured to use a convolutional neural network (CNN) to identify the hand gesture.

3. The eyewear of claim 1 , wherein the electronic processor is configured to identify the hand gesture by matching the hand gesture in the image to a set of hand gestures.

4. The eyewear of claim 3 , further comprising a memory, wherein the set of hand gestures is a library of hand gestures stored in the memory.

5. The eyewear of claim 1 , wherein the hand gesture comprises a static hand gesture.

6. The eyewear of claim 5 , wherein the electronic processor is configured to identify a word from a series of hand gestures.

7. The eyewear of claim 1 , wherein the hand gesture comprises a moving hand gesture.

8. A method of use of eyewear having a frame configured to be worn on a head of a user, a speaker supported by the frame, a camera supported by the frame and configured to generate an image including a hand gesture of a user of the eyewear, and an electronic processor, the electronic processor:
receiving the image including the hand gesture from the camera;
identifying the hand gesture as a sign language; and
generating speech via the speaker that is indicative of the identified hand gesture of the user, wherein the speech is configured to be heard and understood by a person to enable a converation between the user and the person.

9. The method of claim 8 , wherein the electronic processor uses a convolutional neural network (CNN) to identify the hand gesture.

10. The method of claim 8 , wherein the electronic processor is configured to identify the hand gesture by matching the hand gesture in the image to a set of hand gestures.

11. The method of claim 10 , wherein the eyewear comprises a memory, wherein the set of hand gestures is a library of hand gestures stored in the memory.

12. The method of claim 8 , wherein the hand gesture comprises a static hand gesture.

13. The method of claim 12 , wherein the electronic processor identifies a word from a series of hand gestures.

14. The method of claim 8 , wherein the hand gesture comprises a moving hand gesture.

15. A non-transitory computer-readable medium storing program code which, when executed by an electronic processor of eyewear having a frame configured to be worn on a head of a user, a speaker supported by the frame, a camera supported by the frame and configured to generate an image including a hand gesture of a user of the eyewear, is operative to cause the processor to perform the steps of:
receiving the image including the hand gesture from the camera;
identifying the hand gesture as a sign language; and
generating speech via the speaker that is indicative of the identified hand gesture of the user, whrein the speech is configured to be heard and understood by a person to enable a conversation between the user and the person.

16. The non-transitory computer-readable medium of claim 15 , wherein the program code is operative to cause the electronic processor to a convolutional neural network (CNN) to identify the hand gesture.

17. The non-transitory computer-readable medium of claim 15 , wherein the program code is operative to cause the electronic processor to identify the hand gesture by matching the hand gesture in the image to a set of hand gestures.

18. The non-transitory computer-readable medium of claim 17 , wherein the eyewear comprises a memory storing the set of hand gestures.

19. The non-transitory computer-readable medium of claim 15 , wherein the hand gesture comprises a moving hand gesture.

20. The non-transitory computer-readable medium of claim 15 , wherein the program code is operative to cause the electronic processor to identify a word from a series of hand gestures.

Assignees

• Snap Inc

Inventors

• Chan Ryan
• Mills Brent
• Pilipski Eitan
• Pounds Jennica
• Solomon Elliot

Classifications

• G06V40/28 Primary

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06N3/02

  Neural networks · CPC title

• G06V10/95

  structured as a network, e.g. client-server architectures · CPC title

• G10L13/027

  Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08) · CPC title