Electronic device for processing multi-modal input, method for processing multi-modal input and server for processing multi-modal input

Patent metadata

Field	Value
Publication number	US-10191718-B2
Application number	US-201715824331-A
Country	US
Kind code	B2
Filing date	Nov 28, 2017
Priority date	Nov 28, 2016
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

An electronic device is provided. The electronic device includes a housing, a touchscreen display exposed through a first portion of the housing, a microphone disposed at a second portion of the housing, a speaker disposed at a third portion of the housing, a memory disposed inside the housing, a processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory. The memory is configured to store a plurality of application programs, each of which includes a graphic user interface (GUI).

First claim

What is claimed is:

1. An electronic device comprising:
a housing;
a touchscreen display exposed through a first portion of the housing;
a microphone disposed at a second portion of the housing;
a speaker disposed at a third portion of the housing;
a memory disposed inside the housing; and
at least one processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory,
wherein the memory is configured to store a plurality of application programs, each of the application programs including a graphic user interface (GUI), and
wherein the memory is further configured to store instructions that, when executed, cause the at least one processor to:
control the microphone to receive an utterance from a user;
generate or receive text data based on the utterance;
generate or receive a sequence of states in which the electronic device performs a task based on the text data, wherein the sequence of states includes at least two states of at least one of the application programs;
select one of the application programs, based at least partly on the sequence of states;
launch the selected application program;
control to display the GUI of the selected application program on the display;
control to display at least part of a sequence of automatically providing an input into the GUI on the display, based at least partly on the sequence of states;
cause the selected application program to perform an operation that utilizes at least one parameter, using the input as the at least one parameter, depending on the sequence of states; and
control to display a result of the operation on the display.

2. The electronic device of claim 1 , wherein the instructions further cause the at least one processor to download at least one of the plurality of application programs from a server.

3. The electronic device of claim 1 , wherein the instructions are part of a software framework that provides an application programming interface to the plurality of application programs.

4. The electronic device of claim 1 , wherein the instructions further cause the at least one processor to:
control to display, on the display, one or more identifiers, each of which represents a respective one of the plurality of application programs, based at least partly on the text data;
receive a selection of one of the identifiers from the user via the display or the microphone; and
select the one of the application programs, based on the received selection.

5. The electronic device of claim 4 , wherein the identifiers are in a form of at least one of an icon, a symbol, an image, or text.

6. The electronic device of claim 1 , wherein, when the text data lacks a parameter for the selected application program to perform the operation, the instructions further cause the at least one processor to:
control to display a cursor in an input area of the GUI for receiving the parameter;
receive text input for the parameter via the display; and
cause the application program to perform the operation, using the text input.

7. The electronic device of claim 1 , wherein the instructions further cause the at least one processor to:
generate or receive intent of the utterance and the parameter for performing the intent based on the text data; and
generate or receive the sequence of the operation of the selected application program based on the intent and the parameter.

8. The electronic device of claim 7 , wherein the one path rule includes the operation of the selected application program and the parameter utilized to perform the operation.

9. An electronic device comprising:
a housing;
a touchscreen display exposed through a first portion of the housing;
a microphone disposed at a second portion of the housing;
a speaker disposed at a third portion of the housing;
a memory disposed inside the housing; and
at least one processor disposed inside the housing, and electrically connected to the display, the microphone, the speaker, and the memory,
wherein the memory is configured to store a first plurality of application programs and a second plurality of application programs, wherein each of the application programs includes a graphic user interface (GUI), and
wherein the memory is further configured to store instructions that, when executed, cause the at least one processor to:
control the microphone to receive an utterance from a user;
generate or receive text data based on the utterance;
generate or receive a sequence of states in which the electronic device performs a task based on the text data, wherein the sequence of states includes at least two states of at least one of the application programs;
select a first application program from the first plurality of application programs, based at least partly on the sequence of states;
launch the selected first application program;
control to display the GUI of the selected first application program on the display;
cause the selected first application program to perform a first operation that utilizes at least one first parameter, based at least partly on the sequence of states, while displaying at least part of a sequence of performing the first operation on the display;
control to display a result of the first operation on the display;
after displaying the result, control to display one or more identifiers on the display, based at least partly on the sequence of states, wherein each of the identifiers represents a respective one of the second plurality of application programs;
receive a selection of one of the identifiers from the user via the display or the microphone;
select a second application program from the second plurality of application programs, based on the received selection; and
cause the selected second application program to perform a second operation that utilizes at least one second parameter, using at least part of the result of the first operation.

10. The electronic device of claim 9 , wherein the instructions further cause the at least one processor to download at least one of the first plurality of application programs or the second plurality of application programs from a server.

11. The electronic device of claim 9 , wherein the instructions are part of a software framework that provides an application programming interface to the first plurality of application programs and the second plurality of application programs.

12. The electronic device of claim 9 , wherein the instructions further cause the at least one processor to:
after selecting the second application program, control to display the GUI of the selected second application program on the display;
cause the selected second application program to perform the second operation, while displaying at least part of a sequence of performing the second operation on the display; and
control to display a result of the second operation on the display.

13. The electronic device of claim 9 , wherein the identifiers are in a form of at least one of an icon, a symbol, an image, or text.

14. The electronic device of claim 9 , wherein, when the text data lacks a first parameter for the first application program to perform the first operation, the instructions further cause the at least one processor to:
control to display an input area of the GUI of the first application program on the display for receiving the first parameter;
receive a first input for the first parameter via the display or the microphone; and
cause the first application program to perform the operation, using the first input.

15. The electronic device of claim 9 , wherei

Assignees

• Samsung Electronics Co Ltd

Inventors

• Rhee In Jong
• Lee Ji Min
• Kang Sang Ki
• Ku Han Jun
• Park Sung Pa
• Seo Jang Seok
• Song In Wook
• Ahn Won Ick
• Woo Kyoung Gu
• Yi Ji Soo
• Jeon Chang Kyun
• Jaygarl Ho Jun
• Choi Il Hwan
• Hong Yoo Jin
• Kim Ji Hyun
• Yeo Jae Yung

Classifications

• G10L15/265

  Physics · mapped topic

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title