Using human wizards in a conversational understanding system

Patent metadata

Field	Value
Publication number	US-9830039-B2
Application number	US-201313784449-A
Country	US
Kind code	B2
Filing date	Mar 4, 2013
Priority date	Mar 4, 2013
Publication date	Nov 28, 2017
Grant date	Nov 28, 2017

Abstract

A wizard control panel may be used by a human wizard to adjust the operation of a Natural Language (NL) conversational system during a real-time dialog flow. Input to the wizard control panel is detected and used to interrupt/change an automatic operation of one or more of the NL conversational system components used during the flow. For example, the wizard control panel may be used to adjust results determined by an Automated Speech Recognition (ASR) component, a Natural Language Understanding (NLU) component, a Dialog Manager (DM) component, and a Natural Language Generation (NLG) before the results are used to perform an automatic operation within the flow. A timeout may also be set such that when the timeout expires, the conversational system performs an automated operation by using the results shown in the wizard control panel (edited/not edited).

First claim

What is claimed is:

1. A method for using a wizard control panel in a conversational system, comprising:
receiving a user interaction with the conversational system during a dialog flow;
receiving information associated with graphical entities of a user's display;
displaying the wizard control panel including a display of elements that are used for affecting an automatic operation of the conversational system during the dialog flow and a display of a representative state of the user's display for interacting with the user's display by a human wizard;
processing the user interaction to display results for the user interaction within the wizard control panel;
determining receipt of an update to the results in the wizard control panel; and
in response to determining the results are updated in the wizard control panel, submitting updated results to the conversational system, wherein submission of the updated results modifies operation of the conversational system before the automatic operation is processed.

2. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying text that is automatically recognized by an input recognition component of the conversational system within an element of the wizard that is changeable.

3. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying a Natural Language Understanding element of the wizard that is configured for: viewing an intent and changing the intent; displaying a slot and a value and changing the slot and the value; wherein the Natural Language Understanding element is populated using results determined by an input understanding component of the conversational system.

4. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying a Natural Language Generation element of the wizard that is configured for viewing and changing feedback that is populated using results automatically determined by an input understanding component of the conversational system.

5. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying shortcut elements that when selected instruct the conversational system to perform a command.

6. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying a timer that indicates an amount of time left before the conversational system automatically performs actions using results currently shown in the wizard control panel.

7. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying a skip element that when selected instructs the conversational system to ignore the user interaction.

8. The method of claim 1 , wherein displaying the wizard control panel, comprises: displaying a repeat element that when selected instructs the conversational system to request a user to repeat the user interaction; and an unknown element that when selected instructs the conversational system to ignore the user interaction as unknown.

9. The method of claim 1 , wherein submission of the updated results further comprises transmitting a control action to manipulate an entity on the user's display.

10. A computer-readable hardware storage device storing computer-executable instructions that, when executed on a processor, causes the processor to perform a method comprising:
receiving a user interaction with a conversational system during dialog flow processing;
receiving information associated with graphical entities of a user's display;
displaying a wizard control panel including a display of elements that are used for affecting an automatic operation of the conversational system during the dialog flow processing and a display of a representative state of the user's display for interacting with the user's display by a human wizard;
processing the user interaction to display results for the user interaction within the wizard control panel;
displaying a timer that indicates an amount of time left of a predetermined time before the conversational system executes the automatic operation using results currently shown in the wizard control panel;
determining when an input is received into the wizard control panel within the predetermined time, wherein the input updates the results displayed in the wizard control panel; and
in response to determining that the input is received into the wizard control panel, submitting updated results based on the input, wherein the updated results modifies operation of the conversational system before the automatic operation is processed.

11. The computer-readable hardware storage device of claim 10 , wherein displaying the wizard control panel, comprises: displaying text that is automatically recognized by an input recognition component of the conversational system within an element of the wizard that is changeable.

12. The computer-readable hardware storage device of claim 10 , wherein displaying the wizard control panel, comprises: displaying a Natural Language Understanding element of the wizard that is configured for: viewing an intent and changing the intent; displaying a slot and a value and changing the slot and the value; wherein the Natural Language Understanding element is populated using results determined by an input understanding component of the conversational system.

13. The computer-readable hardware storage device of claim 10 , wherein displaying the wizard control panel, comprises: displaying a Natural Language Generation element of the wizard that is configured for viewing and changing feedback that is populated using results automatically determined by an input understanding component of the conversational system.

14. The computer-readable hardware storage device of claim 10 , wherein displaying the wizard control panel, comprises: displaying shortcut elements that when selected instruct the conversational system to perform a command comprising one or more of: a back command, a help command, a home command, a scroll up command, a scroll down command, and a search command.

15. The computer-readable hardware storage device of claim 10 , wherein displaying the wizard control panel, comprises: displaying one or more of: a skip element that when selected instructs the conversational system to ignore the user interaction; a repeat element that when selected instructs the conversational system to request a user to repeat the user interaction; and an unknown element that when selected instructs the conversational system to ignore the user interaction as unknown.

16. The computer-readable storage device of claim 10 , wherein submission of the updated results further comprises transmitting a control action to manipulate an entity on the user's display.

17. A system for using a wizard control panel in a natural language (NL) conversational system, comprising:
at least one processor;
a display; and
a memory operatively connected with the at least one processor, the memory storing instructions, that when executed on the at least one processor, causes the at least one processor to perform operations comprising:
receiving a user interaction with the NL conversational system during processing of a dialog flow;
receiving graphical entities from a user's display;
displaying the wizard control panel comprising a display of a representative state of the user's display for interacting with the user's display by a human wizard;
processing the user interaction to display results for the user interaction within the wizard control panel and display the graphical entities;
displaying a timer that indicates an amount of time l

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Stifelman Lisa
• Hakkani-Tur Dilek
• Heck Larry
• Chinthakunta Madhusudan

Classifications

• G06F3/048 Primary

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title