Augmented reality translation of sign language classifier constructions

Patent metadata

Field	Value
Publication number	US-12307764-B2
Application number	US-202117303313-A
Country	US
Kind code	B2
Filing date	May 26, 2021
Priority date	May 26, 2021
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

A method, computer system, and a computer program product for translating a classifier construction into a graphical representation is provided. The present invention may include observing a classifier handshape by an augmented reality device. The present invention may include analyzing the observed classifier handshape according to an object recognition algorithm to determine a contextual meaning of the classifier handshape. The present invention may include converting the contextual meaning of the observed classifier handshape into a graphical representation. The present invention may include displaying the graphical representation alongside the observed classifier handshape on the augmented reality device.

First claim

What is claimed is:

1. A method for translating a classifier construction into a graphical representation, the method comprising:
observing a classifier handshape by an augmented reality device;
analyzing the observed classifier handshape according to an object recognition algorithm to determine a contextual meaning of the classifier handshape;
converting the contextual meaning of the observed classifier handshape into a graphical representation, wherein converting the contextual meaning into the graphical representation further comprises retrieving a generated image or video imagery from a repository, and based on one or more additional adjectival modifiers of a term in a conversation and based on a context of the conversation associated with the observed classifier handshape, automatically modifying the generated image or video imagery with parameters matching the contextual meaning of the one or more additional adjectival modifiers from the conversation;
displaying the graphical representation, wherein displaying the graphical representation further comprises, superimposing the graphical representation in the augmented reality device.

2. The method of claim 1 , wherein the contextual meaning is based on the observed classifier handshape and a conversational context and is determined utilizing a connected corpus.

3. The method of claim 1 , wherein the graphical representation is an animated Graphics Interchange Forms (GIF).

4. The method of claim 3 , wherein the GIF is retrieved from a corpus by a neural network trained using a plurality of classifier handshapes.

5. The method of claim 1 , wherein the observed classifier handshape is an American Sign Language (ASL) gesture.

6. The method of claim 1 , wherein the object recognition algorithm is a multi -class classifier trained to associate a video and/or a still image with an associated word and to output a label.

7. The method of claim 1 , wherein the graphical representation is overlayed on an existing video feed.

8. A computer system for translating a classifier construction into a graphical representation, comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising:
observing a classifier handshape by an augmented reality device;
analyzing the observed classifier handshape according to an object recognition algorithm to determine a contextual meaning of the classifier handshape;
converting the contextual meaning of the observed classifier handshape into a graphical representation, wherein converting the contextual meaning into the graphical representation further comprises retrieving a generated image or video imagery from a repository, and based on one or more additional adjectival modifiers of a term in a conversation and based on a context of the conversation associated with the observed classifier handshape, automatically modifying the generated image or video imagery with parameters matching the contextual meaning of the one or more additional adjectival modifiers from the conversation;
displaying the graphical representation, wherein displaying the graphical representation further comprises, superimposing the graphical representation in the augmented reality device.

9. The computer system of claim 8 , wherein the contextual meaning is based on the observed classifier handshape and a conversational context and is determined utilizing a connected corpus.

10. The computer system of claim 8 , wherein the graphical representation is an animated Graphics Interchange Forms (GIF).

11. The computer system of claim 10 , wherein the GIF is retrieved from a corpus by a neural network trained using a plurality of classifier handshapes.

12. The computer system of claim 8 , wherein the observed classifier handshape is an American Sign Language (ASL) gesture.

13. The computer system of claim 8 , wherein the object recognition algorithm is a multi-class classifier trained to associate a video and/or a still image with an associated word and to output a label.

14. The computer system of claim 8 , wherein the graphical representation is overlayed on an existing video feed.

15. A computer program product for translating a classifier construction into a graphical representation, comprising:
one or more non-transitory computer-readable storage media and program instructions stored on at least one of the one or more tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:
observing a classifier handshape by an augmented reality device;
analyzing the observed classifier handshape according to an object recognition algorithm to determine a contextual meaning of the classifier handshape;
converting the contextual meaning of the observed classifier handshape into a graphical representation, wherein converting the contextual meaning into the graphical representation further comprises retrieving a generated image or video imagery from a repository, and based on one or more additional adjectival modifiers of a term in a conversation and based on a context of the conversation associated with the observed classifier handshape, automatically modifying the generated image or video imagery with parameters matching the contextual meaning of the one or more additional adjectival modifiers from the conversation;
displaying the graphical representation, wherein displaying the graphical representation further comprises, superimposing the graphical representation in the augmented reality device.

16. The computer program product of claim 15 , wherein the contextual meaning is based on the observed classifier handshape and a conversational context and is determined utilizing a connected corpus.

17. The computer program product of claim 15 , wherein the graphical representation is an animated Graphics Interchange Forms (GIF).

18. The computer program product of claim 17 , wherein the GIF is retrieved from a corpus by a neural network trained using a plurality of classifier handshapes.

19. The computer program product of claim 15 , wherein the observed classifier handshape is an American Sign Language (ASL) gesture.

20. The computer program product of claim 15 , wherein the object recognition algorithm is a multi-class classifier trained to associate a video and/or a still image with an associated word and to output a label.

Assignees

• IBM

Inventors

• Beller Charles E
• Silverstein Zachary A
• Fox Jeremy R
• Decrop Clement

Classifications

• G06V40/28 Primary

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T13/00

  Animation · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V40/113

  Recognition of static hand signs · CPC title