Semiautomated relay method and apparatus

Patent metadata

Field	Value
Publication number	US-2022014623-A1
Application number	US-202117486375-A
Country	US
Kind code	A1
Filing date	Sep 27, 2021
Priority date	Feb 28, 2014
Publication date	Jan 13, 2022
Grant date	—

Abstract

A captioning relay for captioning hearing user (HU) voice signals comprising a plurality of separate captioning resources and a captioning administrator module that receives HU voice signal segments corresponding to a plurality of separate ongoing calls between HUs and AUs and provides the voice signal segments in a first in, first out order to the captioning resources, the administrator module providing each voice signal segment from each call to any one of the captioning resources to be captioned without regard to which captioning resource captioned prior voice signal segments generated during the call and, the administrator module further receiving caption segments back from the captioning resources and providing those captioning segments to AU devices associated with the calls that generated corresponding HU voice signal segments, and wherein the number of captioning resources is less than the number of ongoing calls.

First claim

What is claimed is:

1 . A communication system for enabling communication between an assisted user (AU) and a hearing user (HU) where the HU uses and HU communication device, the system comprising:
an AU communication device including: a display; at least a first processor linked to the display; a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
receiving an HU voice signal from the HU device;
providing the HU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device;
generating first text captions corresponding to the HU voice signal using the ASR engine;
automatically determining whether the generated first text captions meet a first accuracy threshold;
when the first text captions meet the first accuracy threshold, presenting the first text captions via the display;
only when the first text captions fail to meet the first accuracy threshold:
(i) transmitting the HU voice signal corresponding to the first text captions to a remote relay; (ii) receiving second text captions corresponding to at least a subset of the first text captions from the relay; and (iii) presenting the second text captions via the display.

2 . The communication system of claim 1 further including the step of, when the first text captions fail to meet the first accuracy threshold, transmitting the first text captions to the remote relay.

3 . The communication system of claim 2 further including the remote relay, the relay further including a call assistant (CA) workstation including an interface device, a second memory, a second display, and at least a second processor linked to the interface device and the second memory, the second memory having stored thereon software such that, when executed by the at least a second processor, the second processor receives the HU voice signal, presented to the HU voice signal to the CA, presents the first text captions to the CA via the second display, and receives CA error corrections via the interface device to generate the second text captions.

4 . The communication system of claim 3 wherein the first processor generates confidence factors for captioned text and automatically determines accuracy based on the confidence factors.

5 . The communication system of 4 wherein the second processor visually distinguishes low confidence text words in the first text captions presented to the CA on the second display.

6 . The communication system of claim 5 wherein the low confidence text words are visually distinguished via highlighting those words in a distinguishing color.

7 . The communication system of claim 3 wherein the relay further includes a speaker, the second processor providing the HU voice signal to the CA by broadcasting the HU voice signal via the speaker.

8 . The communication system of claim 1 wherein the AU device is a wireless communication device.

9 . The communication system of claim 1 wherein the AU device is a smart phone device.

10 . The communication system of claim 1 wherein the AU device further includes a speaker linked to the first processor and wherein the first processor further performs an operation to broadcast the HU voice signal via the speaker.

11 . The communication system of claim 1 further including the remote relay, the relay further including a call assistant (CA) workstation including an interface device, a second memory, and at least a second processor linked to the interface device and the second memory, the second memory having stored thereon software such that, when the software is executed by the at least a second processor, the at least a second processor receives the HU voice signal, presents the HU voice signal to the CA, generates second text captions associated with the HU voice signal based on CA input via the interface device, and transmits the second text captions to the AU device.

12 . The communication system of claim 11 wherein the relay further includes a microphone, the CA input including the CA revoicing the HU voice signal into the microphone, the second processor running another ASR trained to the CA's voice to generate CA voice captions.

13 . The communication system of claim 11 wherein the CA voice captions are presented on a display screen to the CA, the second processor further receiving CA error corrections via the interface device and generating the second text captions.

14 . The communication system of claim 1 wherein the first text captions are presented via the display regardless of whether or not the first text captions meet the first accuracy threshold and prior to receiving the second text captions.

15 . The communication system of claim 14 wherein the second text captions are used to perform in line correction to the first text captions presented via the display.

16 . The communication system of claim 15 wherein any corrections to the first text captions on the display are visually distinguished from text captions that are corrected.

17 . The communication system of claim 1 wherein the AU device further includes a microphone for capturing an AU voice signal, the first processor further programmed to perform operations including receiving the AU voice signal, providing the AU voice signal to an automated speech recognition (ASR) engine operated by the at least a first processor on the AU device to generate AU voice signal captions and presenting the AU voice signal captions via the display.

18 . The communication system of claim 17 further including the step of, when the first text captions fail to meet the first accuracy threshold, transmitting the first text captions to the remote relay.

19 . The communication system of claim 17 wherein the first text captions are transmitted to the remote relay and the AU voice signal captions are not transmitted to the remote relay.

20 . The communication system of claim 17 wherein the AU voice signal captions are visually distinguished from the text captions associated with the HU voice signal as they are presented on the display.

21 . The communication system of claim 20 wherein the AU voice signal captions are presented in a first column on the display and the text captions are presented in a second column on the display.

22 . The communication system of claim 1 wherein the AU device further includes a selectable input that, when selected, causes the first processor to start transmitting the HU voice signal to the relay for call assistant assisted captioning services.

23 . The communication system of claim 22 wherein, upon selection of the selectable input, the first processor also transmits the first captions to the relay to be presented to a call assistant.

24 . A communication system including an assisted user's (AU's) captioning device for use by an assisted user when communicating with a hearing user (HU) using a hearing user's (HU's) device, the AU's device comprising:
a display; at least a first processor linked to the display; a first memory having stored thereon software such that, when the software is executed by the at least a first processor, the at least a first processor generates text captions from speech data by performing operations including:
receiving an HU voice signal from the HU device;

Assignees

• Ultratec Inc

Inventors

• Engelke Robert M
• Colwell Kevin R
• Engelke Christopher

Classifications

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• H04M1/72478

  for hearing-impaired users · CPC title

• H04M2201/60

  Medium conversion · CPC title

• H04M1/72433

  for voice messaging, e.g. dictaphones (for answering incoming calls H04M1/64) · CPC title

• G10L15/01

  Assessment or evaluation of speech recognition systems · CPC title