Semiautomated relay method and apparatus

Patent metadata

Field	Value
Publication number	US-11741963-B2
Application number	US-201816147029-A
Country	US
Kind code	B2
Filing date	Sep 28, 2018
Priority date	Feb 28, 2014
Publication date	Aug 29, 2023
Grant date	Aug 29, 2023

Abstract

A method and system for providing captioned telephone service, the method comprising the steps of, initiating a first captioned telephone service call, during the first captioned telephone service call, creating a first set of captions using a call assistant, simultaneous with creating the first set of captions using a call assistant, creating a second set of captions using an automated speech recognition engine, comparing the first set of captions and the second set of captions using a scoring algorithm based on errors between the first and second sets of captions to generate a score for the second set of captions, in response to the score being within a predetermined threshold range, continuing the call using only the automated speech recognition engine to generate text and in response to the score being outside of the predetermined threshold range, continuing the call using a call assistant to generate captions.

First claim

What is claimed is:

1. A method for providing captioned telephone service, the method comprising:
initiating a first captioned telephone service call;
during the first captioned telephone service call, creating a first set of captions using a call assistant;
simultaneous with creating the first set of captions using a call assistant, creating a second set of captions using an automated speech recognition (ASR) engine;
comparing the first set of captions and the second set of captions to identify errors in the second set of captions;
in response to the identified errors in the second set of captions being less than a threshold level, continuing the call using only the automated speech recognition engine to generate the second set of captions; and
in response to the identified errors in the second set of captions being greater than a threshold level, continuing the call using a call assistant to generate the first set of captions.

2. The method of claim 1 , further comprising, in response to the identified errors in the second set of captions being less than the threshold level, storing an indication of accuracy of the ASR engine in a database.

3. The method of claim 2 further comprising:
initiating a second captioned telephone service call;
routing the second captioned telephone service call to a call assistant if the identified errors in the second set of captions exceeds the threshold level; and
routing the second captioned telephone service call to the automated speech recognition engine if the identified errors in the second set of captions are below the threshold level.

4. The method of claim 3 wherein the first captioned telephone service call is between an assisted user and a hearing user, the method further including upon initiation of the second captioned telephone service call, identifying the hearing user and identifying an accuracy of the ASR engine for the identified hearing user.

5. The method of claim 4 wherein the second captioned telephone service call is initially routed to a call assistant during a period while the hearing user is being identified.

6. The method of claim 5 wherein the second captioned telephone service call is also initially routed to an automated speech recognition engine so that captions of the hearing user's voice signal are simultaneously generated by each of the call assistant and the automated speech recognition engine.

7. The method of claim 6 wherein a voice to text model is used by the automated speech recognition engine.

8. The method of claim 7 wherein differences between the captions generated by the call assistant and the automated speech recognition engine are used to train the (ASR) engine to the hearing user's voice signal thereby increasing the probability of success of the ASR engine until the probability of success exceeds a threshold level.

9. The method of claim 2 wherein the indication of the accuracy of the ASR engine indicates a likelihood that the automated speech recognition engine will generate text captions that exceed the acceptable threshold level under similar telephone service call circumstances.

10. The method of claim 1 further including using a scoring algorithm-based on errors between the first and second sets of captions to generate a score for the second set of captions, the step of continuing the call using only the automated speech recognition engine to generate text includes, in response to the score being within a predetermined threshold range, continuing the call using only the automated speech recognition engine to generate text, and, wherein, the step of continuing the call using a call assistant to generate captions includes, in response to the score being outside of the predetermined threshold range, continuing the call using a call assistant to generate captions.

11. The method of claim 10 wherein the score is within the predetermined threshold range when an error rate is less than a threshold level and is outside the predetermined threshold range when the error rate is greater than the threshold level.

12. The method of claim 10 wherein the scoring algorithm uses accuracy of the second set of captions over a duration of time.

13. The method of claim 10 wherein the scoring algorithm uses detected line noise.

14. The method of claim 10 wherein the scoring algorithm uses at least one audio metric.

15. The method of claim 10 wherein the score indicates a percent match between the automated speech recognition engine captions and the call assistant captions.

16. The method of claim 10 wherein the scoring algorithm uses a percent accuracy of the second set of captions over a most recent duration of time.

17. The method of claim 10 wherein the scoring algorithm uses a percent accuracy over a most recent number of words in a received voice signal.

18. The method of claim 10 wherein the scoring algorithm utilizes confidence statistics to score the second set off captions.

19. The method of claim 10 wherein the scoring algorithm utilizes detected line noise.

20. The method of claim 10 wherein the scoring algorithm factors in at least one audio metric related to the first call.

21. The method of claim 1 wherein the first captioned telephone service call is an internet protocol captioned telephone service call.

22. The method of claim 1 , in which the step of comparing the first set of captions and the second set of captions is performed substantially in real time during the first captioned telephone service call.

23. The method of claim 1 wherein the first call is between an assisted user and a hearing user and wherein only the hearing user's voice signals are captioned.

24. The method of claim 1 further comprising:
in response to the level of identified errors in the second set off captions relative to the threshold level, storing an indication of accuracy of the ASR engine while captioning the first caption telephone service call;
initiating a second captioned telephone service call;
retrieving the indication of the accuracy of the ASR engine while captioning the first caption telephone service call;
using, on the second captioned telephone service call, a call assistant or the automated speech recognition engine to generate captions based on the indication of the accuracy of the ASR engine while captioning the first caption telephone service call.

25. The method of claim 24 wherein the indication of the accuracy of the ASR engine includes a percent word match between the first and second sets of captions.

26. The method of claim 1 wherein the step of continuing the call using only the automated speech recognition engine to generate the second set of captions includes using the automated speech recognition engine to generate initial captions and providing the initial captions to a call assistant for error correction.

27. The method of claim 1 wherein the step of creating a first set of captions using a call assistant includes the call assistant examining captions generated by the automated speech recognition engine and making error corrections to the automated speech recognition captions.

28. The method of claim 1 wherein the step of creating a first set of captions using a call assistant includes the call assistant listening to at least a portion of voice signals from the first call and manually transcribing that voice signal to text.

Assignees

• Ultratec Inc

Inventors

• Engelke Robert M
• Colwell Kevin R
• Engelke Christopher

Classifications

• H04M3/42391 Primary

  where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf · CPC title

• G10L25/48

  specially adapted for particular use · CPC title

• G10L15/26 Primary

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L15/01

  Assessment or evaluation of speech recognition systems · CPC title

• H04M1/2475

  for a hearing impaired user · CPC title