In-call translation

Patent metadata

Field	Value
Publication number	US-9614969-B2
Application number	US-201514622311-A
Country	US
Kind code	B2
Filing date	Feb 13, 2015
Priority date	May 27, 2014
Publication date	Apr 4, 2017
Grant date	Apr 4, 2017

Abstract

The disclosure pertains to a communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language. A translation procedure is performed on call audio of the call to generate an audio translation of the source user's speech in the target language for outputting to the target user. A notification is outputted to the target user to notify the target user of a change in the behavior of the translation procedure, the change relating to the generation of the translation.

First claim

The invention claimed is:

1. A computer-implemented method performed in a communication system, the communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language, the method comprising:
receiving call audio of the call, the call audio comprising speech of the source user in the source language;
performing, by a speech translator module, an automatic translation procedure on the call audio to generate an audio translation of the source user's speech in the target language for outputting to the target user; and
signalling, by the speech translator module, a change in behaviour of the automatic translation procedure, the change relating to the generation of the automatic translation, and thereby causing a notification to be outputted to the target user to notify the target user of the change, the notification corresponding to the change in behaviour of the translation procedure and including a synthetic video embodied as a visual action by an animated avatar mimicking visual cues of a human.

2. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering a listening state, in which it is currently awaiting future speech activity by the source user during a current interval of speech inactivity by the source user.

3. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering a passive translation state responsive to the source user commencing a period of speech activity, in which the translation procedure is monitoring current speech activity by the source user in the call audio.

4. The computer-implemented method according to claim 1 wherein the change in behaviour is the translation procedure entering an active translation state responsive to the source user finishing an interval of speech activity, in which the translation procedure is currently generating an audio translation of the source user's speech in that interval to be outputted when that generating is complete.

5. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering an outputting state responsive to the translation procedure completing generation of an audio translation of the source user's speech during a preceding interval of source user speech activity, in which that generated audio translation is currently being outputted by the translation procedure for outputting to the target user.

6. The computer-implemented method according to claim 1 wherein the change in the behaviour is the translation procedure entering an error state responsive to the procedure encountering an error in generating the translation.

7. The computer-implemented method according to claim 1 wherein the translated audio is transmitted via a communication network as it is generated to a target device of the target user for outputting via one or more audio output components of that device as it is received.

8. The computer-implemented method according to claim 1 wherein the notification comprises a visual notification for displaying at a target user device of the target user and/or an audio notification for playing out at the target user device and/or a tactile notification outputted by actuating a mechanical component of the target user device.

9. A computer system for use in a communication system, the communication system for effecting a voice or video call between at least a source user speaking a source language and a target user speaking a target language, the computer system comprising:
one or more audio output components available to the target user;
a translation output component configured to output an audio translation of the source user's speech in the target language to the target user via the audio output components, the translation generated by performing an automatic translation procedure on call audio of the call which comprises speech of the source user in the source language; and
a notification output component configured to output a notification to the target user to notify the target user of a change in behaviour of the translation procedure, the change relating to the generation of the translation and the notification including a synthetic video embodied as a visual action by an animated avatar mimicking visual cues of a human.

10. The computer system according to claim 9 wherein the call audio comprises speech of the source user in the source language during intervals of source user speech activity interspersed with intervals of speech inactivity in which the source user is not speaking;
wherein, for at least one interval of source user speech activity, the translation output component is configured to output via the audio output components an audio translation of the source user's speech during that interval, and wherein the notification output component is configured to output the notification when the outputting of that translation has substantially finished to indicate that the target user is free to respond to the source user.

11. The computer system according to claim 9 wherein the computer system is embodied by a target user device of the target user or by a combination the target user device and at least one other computer device to which the target user device is connected via a communication network.

12. The computer system according to claim 9 comprising:
an input configured to receive a signal signalling the change in the behaviour of translation procedure; and
a notification generation component configured to generate the notification in dependence on the received signal.

13. The computer system according to claim 12 wherein the notification output component is configured to generate output-related information defining the manner in which the notification is to be outputted to the target user; and
wherein the notification generation component is configured to generate the notification in dependence on the output-related information.

14. The computer system according to claim 13 comprising a display available to the target user, wherein the synthetic video embodied as the visual action is displayed on the display and the output-related information comprises related layout information.

15. The computer system according to claim 14 wherein the notification generation component is configured to generate the synthetic video embodied as the visual action and embodying the notification, the synthetic video generated in dependence on the layout information.

16. The computer system according to claim 15 wherein the animated avatar mimicking visual cues of a human is controlled in dependence on the layout information.

17. A computer program product comprising computer code stored on a computer readable storage device configured when executed on a processor to cause operations of:
establishing a voice or video call between at least a source user speaking a source language and a target user speaking a target language;
outputting an audio translation of the source user's speech in the target language to the target user, the translation generated by performing an automatic translation procedure on call audio of the call which comprises speech of the source user in the source language; and
outputting a notification to the target user to notify the target user of a change in behaviour of the translation procedure, the change relating to the generation of the translation and the notification including a synthetic vi

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Aue Anthony
• Menezes Arul A
• Lindblom Jonas Nils
• Furesjö Fredrik
• Greborio Pierre P N

Classifications

• G06F40/58 Primary

  Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation · CPC title

• H04M3/42

  Systems providing special services or facilities to subscribers (specially adapted for wireless communication networks H04W4/00) · CPC title

• H04M2201/50

  Telephonic communication in combination with video communication · CPC title

• H04M2242/12

  Language recognition, selection or translation arrangements · CPC title

• H04M2203/2061

  Language aspects · CPC title