Automatically adapting user interfaces for hands-free interaction

Patent metadata

Field	Value
Publication number	US-10496753-B2
Application number	US-201113250947-A
Country	US
Kind code	B2
Filing date	Sep 30, 2011
Priority date	Jan 18, 2010
Publication date	Dec 3, 2019
Grant date	Dec 3, 2019

Abstract

A user interface for a system such as a virtual assistant is automatically adapted for hands-free use. A hands-free context is detected via automatic or manual means, and the system adapts various stages of a complex interactive system to modify the user experience to reflect the particular limitations of such a context. The system of the present invention thus allows for a single implementation of a complex system such as a virtual assistant to dynamically offer user interface elements and alter user interface behavior to allow hands-free use without compromising the user experience of the same system for hands-on use.

First claim

What is claimed is:

1. A computer-implemented method for adapting a user interface on a computing device having at least one processor, comprising:
performing, at the computing device, a plurality of steps including:
detecting whether a hands-free context is active;
prompting a user for an input;
receiving a user input comprising natural language information;
interpreting the received user input to derive a representation of a user intent, wherein the interpreting of the received user input comprises:
generating a plurality of candidate interpretations based on the received user input,
determining the representation of the user intent based on the plurality of candidate interpretations;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of the user intent;
executing the at least one task using the at least one parameter, to derive a result;
in accordance with the derived result,
paraphrasing at least a portion of the user input in a spoken form;

generating speech using a plurality of voices to differentiate paraphrased user input from other spoken output; and
providing an audio output of the generated speech;
wherein, responsive to a detection that the computing device is in the hands-free context, the user interface is adapted to display a subset of user-interaction mechanisms displayed with the hands-free context being inactive, the subset including at least one user-interaction mechanism.

2. The method of claim 1 , wherein at least two interaction modes are available for user interaction with the computing device, and wherein:
responsive to the detection that the computing device is in the hands-free context, at least one of the steps of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech is performed using a first interaction mode adapted to hand s-free operation; and
responsive to the detection that the computing device is not in the hands-free context, at least one of the steps of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech is performed using a second interaction mode not adapted to hands-free operation.

3. The method of claim 1 , wherein detecting whether a hands-free context is active comprises detecting a condition indicating a limitation in at least one of:
a ability of the user to view visual output presented by the computing device;
a ability of the user to interact with a graphical user interface presented by the computing device;
a ability of the user to use a physical component of the computing device;
a ability of the user to perform a touch input on the computing device;
a ability of the user to activate a switch on the computing device; and
a ability of the user to use a keyboard on the computing device.

4. The method of claim 1 , wherein prompting the user for the input comprises:
responsive to the detection that the computing device is not in the hands-free context, prompting the user via a first output mode not adapted to the hands-free context; and
responsive to the detection that the computing device is in the hands-free context, prompting the user via a second output mode adapted to the hands-free context.

5. The method of claim 4 , wherein:
prompting the user via the first output mode comprises prompting the user via a visual output mode; and
prompting the user via the second output mode comprises prompting the user via an auditory output mode.

6. The method of claim 5 , wherein:
prompting the user via the visual output mode comprises displaying a prompt on a display screen; and
prompting the user via the auditory output mode comprises outputting a spoken prompt.

7. The method of claim 1 , wherein interpreting the received user input comprises:
responsive to the detection that the computing device is in the hands-free context, interpreting the received user input using a vocabulary associated with hands-free operation.

8. The method of claim 1 , wherein identifying at least one task and at least one parameter for the task comprises, responsive to the detection that the computing device is in the hands-free context, performing at least one task flow identification step associated with hands-free operation.

9. The method of claim 8 , wherein performing the at least one task flow identification step associated with the hands-free operation comprises prompting the user to review and confirm entered content via a speech-based interface.

10. The method of claim 8 , wherein performing the at least one task flow identification step associated with the hands-free operation comprises performing the at least one task flow step using an auditory output.

11. The method of claim 8 , wherein performing the at least one task flow identification step associated with the hands-free operation comprises performing the at least one task flow step selected from a limited group of available task flow steps configured for the hands-free context.

12. The method of claim 1 , wherein paraphrasing at least a portion of the user input in the spoken form comprises combining a dialog template with at least one item of personal data.

13. The method of claim 1 , wherein detecting whether or not a hands-free context is active comprises at least one of:
receiving a user input specifying a hands-free context;
receiving data from at least one sensor indicating an environmental condition associated with a hands-free context;
detecting a connection of a peripheral device associated with a hands-free context;
detecting a disconnection of a peripheral device not associated with a hands-free context;
detecting communication with an onboard system of a vehicle;
detecting a current location; and
detecting a current speed.

14. The method of claim 1 , wherein:
prompting the user for the input comprises prompting the user via a conversational interface; and
receiving the user input comprises receiving the user input via the conversational interface.

15. The method of claim 1 , wherein receiving the user input comprises:
receiving a spoken input; and
converting the spoken input to a text representation.

16. The method of claim 1 , wherein the computing device comprises at least one of:
a telephone;
a smartphone;
a tablet computer;
a laptop computer;
a personal digital assistant;
a desktop computer;
a kiosk;
a consumer electronic device;
a consumer entertainment device;
a music player;
a camera;
a television;
an electronic gaming unit; and
a set-top box.

17. The method of claim 1 , wherein the user interface is further adapted to display a visual output corresponding to the audio output of the speech.

18. A computer program product for interpreting a user input to perform a task on a computing device having at least one processor, comprising:
a non-transitory computer-readable storage medium; and
computer program code, encoded on the medium, configured to cause the at least one processor of the computing device to perform a plurality of steps including:
detecting whether a hands-free context is active;
prompting a user for an input;
receiving the user input comprising natural language information;
interpreting the received user input to derive a representation of a user intent, wherein the interpreting of the rec

Assignees

• Gruber Thomas Robert
• Saddler Harry J
• Apple Inc

Inventors

• Gruber Thomas Robert
• Saddler Harry J

Classifications

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F17/279 Primary

  Physics · mapped topic