Video relay service, communication system, and related methods for performing artificial intelligence sign language translation services in a video relay service environment

Patent metadata

Field	Value
Publication number	US-10176366-B1
Application number	US-201715801190-A
Country	US
Kind code	B1
Filing date	Nov 1, 2017
Priority date	Nov 1, 2017
Publication date	Jan 8, 2019
Grant date	Jan 8, 2019

Abstract

Video relay services, communication systems, and methods are disclosed herein. The video relay service for providing automatic translation services during a real-time communication session. The video relay service comprises at least one server configured to receive a video stream containing sign language content from a video communication device associated with a hearing-impaired user; automatically translate the sign language content into a verbal language translation during the real-time communication session without assistance of a human sign language interpreter; and transmit the verbal language translation to a far-end communication device associated with a hearing-capable user during the real-time communication session.

First claim

What is claimed is:

1. A video relay service for providing automatic translation services during a real-time communication session, the video relay service comprising:
at least one server configured to:
receive a video stream containing sign language content from a video communication device associated with a hearing-impaired user;
automatically translate the sign language content into a verbal language translation during the real-time communication session without assistance of a human sign language interpreter by comparing data chunks including multiple frames of the video stream with stored video files in an artificial intelligence (AI) database; and
transmit the verbal language translation to a far-end communication device associated with a hearing-capable user during the real-time communication session.

2. The video relay service of claim 1 , wherein the at least one server is configured to transmit the verbal language translation as synthesized audio.

3. The video relay service of claim 1 , wherein the at least one server configured to receive new entries to add stored video files to the AI database from the video communication device that are available to other users during subsequent communication sessions facilitated by the video relay service.

4. The video relay service of claim 1 , wherein the at least one server is further configured to:
record portions of the communication sessions facilitated by the video relay service; and
add new video files to the stored video files to the AI database generated from the recorded portions of the communication sessions by a training station that are available to other users during subsequent communication sessions facilitated by the video relay service.

5. The video relay service of claim 1 , wherein the at least one server is further configured to:
receive an audio stream from the far-end communication device during the real-time communication session;
automatically translate voice content from the audio stream into another language translation during the real-time communication session; and
transmit the another language translation to the video communication device associated with the hearing-impaired user during the real-time communication session.

6. The video relay service of claim 5 , wherein the another language translation is first generated in a textual format generated by a speech-to-text voice recognition system.

7. The video relay service of claim 6 , wherein the at least one server is configured to transmit the another language translation in the textual format.

8. The video relay service of claim 6 , wherein the at least one server is configured to transmit the another language translation in simulated avatar format depicting the another language translation in sign language without assistance of a human sign language interpreter.

9. The video relay service of claim 6 , wherein the at least one server is configured to transmit the another language translation in both the textual format and a simulated avatar format.

10. The video relay service of claim 6 , wherein the at least one server is configured to transmit the verbal language translation to back to the video communication device associated with a hearing-impaired user during the real-time communication session.

11. The video relay service of claim 6 , wherein the verbal language translation and the another is language translation are the same language.

12. The video relay service of claim 6 , wherein the verbal language translation and the another is language translation are different languages.

13. A method of performing automated translation services during a real-time communication session, the method comprising:
establishing a communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine;
automatically generating, with the AI translation engine, a first translation of sign language content from a video stream from the video communication device without a human sign language interpreter performing the sign language translation for the communication session by dividing the video stream into data chunks of different lengths depending on phrases detected in the sign language content, and comparing the data chunks to stored video files in an AI translation database; and
transmitting the first translation from the AI translation engine to the far-end communication device during the real-time communication session.

14. The method of claim 13 , further comprising:
automatically generating, with the AI translation engine, a second translation of voice content from an audio stream from the far-end communication device; and
transmitting the second translation from the AI translation engine to the video communication device during the real-time communication session.

15. The method of claim 13 , wherein transmitting the first translation from the AI translation engine to the far-end communication device includes synthesized audio of the first translation.

16. The method of claim 13 , further comprising transmitting the first translation from the AI translation engine to the video communication device during the real-time communication session.

17. The method of claim 16 , wherein transmitting the first translation from the AI translation engine to the video communication device includes textual content of the first translation.

18. A video communication system comprising:
a plurality of video communication devices configured for hearing-impaired users to engage in communication sessions with hearing-capable users; and
a video relay service configured to:
establish communication sessions between video communication devices associated with hearing-impaired users and far-end communication device associated with hearing-capable users;
automatically generate translations of sign language content from a video stream from a corresponding video communication device associated with a hearing-impaired user during real-time communication sessions without a human sign language interpreter performing the sign language translation for the communication session by comparing data chunks including multiple frames of the video stream with stored video files in an artificial intelligence (AI) database of an AI translation engine;
transmit the translation from the AI translation engine to a corresponding far-end communication device associated with a hearing-capable user;
automatically generate with the AI translation engine, a second translation of voice content from an audio stream from the corresponding far-end communication device; and
transmit the second translation from the AI translation engine to the video communication device during the real-time communication session.

19. The video relay service of claim 1 , wherein the at least one server is configured to detect a pause between a word or phrase to extract the data chunks from the video stream for comparison with the stored video files in the AI database.

20. The video relay service of claim 1 , wherein a size of the data chunks are hard coded by the at least one server.

21. The video relay service of claim 1 , wherein the at least one server is configured to determine the size of the data chunks based, at least in part, on determining a signing speed for the hearing-impaired user.

22. The video relay service of claim 1 , wherein the at

Assignees

• Sorenson Ip Holdings Llc

Inventors

• Maxwell Conrad A

Classifications

• H04N21/4223 Primary

  Cameras (H04N23/00 takes precedence) · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• H04N21/42203

  sound input device, e.g. microphone · CPC title

• H04N21/234336

  by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text · CPC title