Semiautomated relay method and apparatus

Patent metadata

Field	Value
Publication number	US-10389876-B2
Application number	US-201514632257-A
Country	US
Kind code	B2
Filing date	Feb 26, 2015
Priority date	Feb 28, 2014
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

A system for providing voice-to-text captioning service comprising a relay processor that receives voice messages generated by a hearing user during a call, the processor programmed to present the voice messages to a first call assistant, a call assistant device used by the first call assistant to generate call assistant generated text corresponding to the hearing user's voice messages, the processor further programmed to run automated voice-to-text transcription software to generate automated text corresponding to the hearing user's voice messages, use the call assistant generated text, the automated text and the hearing user's voice messages to train the voice-to-text transcription software to more accurately transcribe the hearing user's voice messages to text, determine when the accuracy exceeds an accuracy requirement threshold, during the call and prior to the automated text exceeding the accuracy requirement threshold, transmitting the call assistant generated text to the assisted user's device for display to the assisted user and subsequent to the automated text exceeding the accuracy requirement threshold, transmitting the automated text to the assisted user's device for display to the assisted user.

First claim

What is claimed is:

1. A system for providing voice-to-text captioning service for a hearing impaired assisted user using an assisted user's device to communicate with a hearing user that uses a hearing user's device, the system comprising:
a relay processor that receives voice messages generated by the hearing user during a call between the assisted user and the hearing user, the relay processor programmed to present the voice messages to a first call assistant;
a call assistant device used by the first call assistant to generate call assistant generated text corresponding to the hearing user's voice messages;
the relay processor further programmed to:
run automated voice-to-text transcription software to generate automated text corresponding to the hearing user's voice messages;
use the call assistant generated text, the automated text and the hearing user's voice messages to train the voice-to-text transcription software to more accurately transcribe the hearing user's voice messages to text;
determine when the accuracy of the automated text exceeds an accuracy requirement threshold;
during the call and prior to the automated text exceeding the accuracy requirement threshold, transmitting the call assistant generated text to the assisted user's device for display to the assisted user; and
subsequent to the automated text exceeding the accuracy requirement threshold, transmitting the automated text to the assisted user's device for display to the assisted user.

2. The system of claim 1 wherein, subsequent to the automated text exceeding the accuracy requirement threshold, the relay processor stops presenting the hearing user's voice messages to the first call assistant so that the first call assistant can transcribe a different hearing user's voice messages.

3. The system of claim 2 further including presenting the text received at the assisted user's device to the assisted user via a device display screen, when automated text from the relay is presented via the display screen, monitoring for an assistance request via the assisted users device and, when an assistance request is received, presenting the hearing user's voice messages to another call assistant to again generate call assistant generated text, transmitting the call assistant generated text to the assisted user's device for display and halting transmission of the automated text to the assisted user's device.

4. The system of claim 3 wherein the step of presenting the hearing user's voice messages to a call assistant after the assistance request has been received includes, determining if the first call assistant is available to handle the hearing user's voice messages and, if the first call assistant is available, presenting the hearing user's voice messages to the first call assistant to be transcribed.

5. The system of claim 3 wherein an assistance request is initiated by selecting a help button of the assisted user's device.

6. The system of claim 2 further including presenting the text received at the assisted user's device to the assisted user via a device display screen, when automated text from the relay is presented via the display screen, also presenting a virtual help button via the display that is selectable by a system user to request captioning from a call assistant.

7. The system of claim 2 further including presenting the call assistant generated text received at the assisted user's device to the assisted user via a device display screen and presenting an indication of a delay in text transcription on the device display screen.

8. The system of claim 7 wherein the indication of a delay includes a number of seconds indicator indicating the duration of an instantaneous delay.

9. The system of claim 7 wherein the indication of a delay includes a representation of words uttered by a hearing user that has yet to be transcribed to text at the end of a current string of displayed text wherein a length dimension of the representation of words uttered is associated with the duration of a current delay in transcription to text.

10. The system of claim 7 further including the step of presenting a re-sync option via the assisted user's device display, detecting selection of the re-sync option and, in response to selection of the re-sync option, skipping transcription of at least a portion of the hearing user's voice message so that the transcription process continues on a more current segment of the hearing user's voice message.

11. The system of claim 10 wherein, in response to selection of the re-sync option, the system skips to a pre-defined duration period that precedes the time when the re-sync selection was made so that automatically generated text is presented as fill in text for the hearing user's voice message that corresponds to the pre-defined duration period up to a current time.

12. The system of claim 11 wherein the fill in text is visually distinguished from other text presented on the assisted user's device display.

13. The system of claim 1 further including the steps of, subsequent to the automated text exceeding the accuracy requirement threshold, presenting the automated text and the hearing user's voice messages to a second call assistant for correction and, after the second call assistant corrects errors in the automated text, transmitting the corrections to the assisted user's device.

14. The system of claim 13 wherein the assisted user's device presents the corrections on the assisted user's device display screen.

15. The system of claim 14 wherein the assisted user's device presents the corrections by replacing errors in the text on the assisted user's device display screen with corrected blocks of text.

16. The system of claim 1 further including the steps of, after at least some training of the voice-to-text software, storing a voice model for the hearing user in a database to be used during subsequent communications between the hearing user and the assisted user.

17. The system of claim 16 wherein each stored voice model is associated with a hearing user's device identifier in the database so that the next time the hearing user's device is linked to the assisted user's device to facilitate a communication, the device identifier can be used to identify the voice model associated with the hearing user.

18. The system of claim 17 wherein more than one voice model is associated with at least one of the device identifiers in the database.

19. The system of claim 18 wherein a voice profile for identifying a specific voice is stored in the database for each of the more than one voice models associated with the at least one of the device identifiers.

20. The system of claim 1 wherein the hearing user's voice messages received at the relay processor are received from the assisted user's device.

21. The system of claim 1 wherein the relay processor trains the voice-to-text transcription software by comparing the call assistant generated text to the automated text to identify errors in the automated text and then adjusts the software to avoid the same error subsequently.

22. The system of claim 1 wherein the relay processor determines when the accuracy of the automated text exceeds an accuracy requirement threshold by comparing the automated text to the call assistant generated text to generate an accuracy value for the automated text and then compares the accuracy value to the accuracy requirement.

23. The system of claim 1 wherein the call assistant generates

Assignees

• Ultratec Inc

Inventors

• Engelke Robert M
• Colwell Kevin R
• Engelke Christopher R

Classifications

• G10L15/06

  Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice (G10L15/14 takes precedence) · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• H04M2201/40

  using speech recognition · CPC title

• H04W4/16

  Communication-related supplementary services, e.g. call-transfer or call-hold · CPC title

• G10L2015/0636

  Threshold criteria for the updating · CPC title