Electronic apparatus for processing user utterance for controlling an external electronic apparatus and controlling method thereof

Patent metadata

Field	Value
Publication number	US-10824392-B2
Application number	US-201816036112-A
Country	US
Kind code	B2
Filing date	Jul 16, 2018
Priority date	Jul 21, 2017
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

An electronic apparatus is provided. The electronic device includes a housing, a touch screen display disposed inside the housing and exposed through a first portion of the housing, a microphone disposed inside the housing and exposed through a second portion of the housing, a communication circuit disposed inside the housing, a processor disposed inside the housing and electrically connected to the touch screen display, the microphone, and the communication circuit, and a memory disposed in the housing and electrically connected with the processor.

First claim

The invention claimed is:

1. An electronic apparatus comprising:
a housing;
a touch screen display disposed inside the housing and exposed through a first portion of the housing;
a microphone disposed inside the housing and exposed through a second portion of the housing;
a communication circuit disposed inside the housing;
at least one processor disposed inside the housing and electrically connected to the touch screen display, the microphone, and the communication circuit; and
a memory disposed in the housing and electrically connected with the at least one processor,
wherein the memory stores an instruction that, when executed, causes the at least one processor to:
receive a user input through at least one of the touch screen display or the microphone, wherein the user input includes a request for performing a first task using a first external electronic apparatus and a second task dependent on the first task using a second external electronic apparatus,
transmit first data associated with the user input to an external server through the communication circuit,
receive a response including information about a first sequence of states for the first external electronic apparatus to perform the first task and a second sequence of states for the second external electronic apparatus to perform the second task, from the external server through the communication circuit,
transmit a first command that allows the first external electronic apparatus to have the first sequence of states to the first external electronic apparatus, and

transmit a second command that allows the second external electronic apparatus to have the second sequence of states to the second external electronic apparatus.

2. The electronic apparatus of claim 1 ,
wherein the first command includes the first sequence of states, and
wherein the second command includes the second sequence of states.

3. The electronic apparatus of claim 1 , wherein the memory further stores an instruction that, when executed, further causes the at least one processor to:
obtain second data by allowing the first external electronic apparatus to have the first sequence of states based on the first command, and
when the second external electronic apparatus performs an operation based on the second sequence of states, transmit the second data together with the second command to the second external electronic apparatus to use the second data based on the second command.

4. The electronic apparatus of claim 3 , wherein, when the first external electronic apparatus and the second external electronic apparatus are included in a local network, the first command allows the first external electronic apparatus to directly transmit the second data to the second external electronic apparatus.

5. The electronic apparatus of claim 1 , wherein the memory further stores an instruction that, when executed, further causes the at least one processor to:
display the first sequence of states and the second sequence of states, which respectively operate based on the first command and the second command, in the touch screen display.

6. The electronic apparatus of claim 1 , wherein the memory further stores an instruction that, when executed, further causes the at least one processor to:
when the first external electronic apparatus and the second external electronic apparatus complete an operation based on the first sequence of states and the second sequence of states, receive second information indicating that the operation has been completed from the first external electronic apparatus and the second external electronic apparatus.

7. The electronic apparatus of claim 6 , wherein the response includes information about a state for verifying that the operation has been completed.

8. The electronic apparatus of claim 1 , wherein the memory further stores an instruction that, when executed, further causes the at least one processor to:
determine whether a first operation of the second external electronic apparatus that is based on the second sequence of states is dependent on a second operation of the first external electronic apparatus that is based on the first sequence of states; and
when the first operation is dependent on the second operation, transmit the second command to the second external electronic apparatus after verifying that the second operation is completed.

9. A device for processing a user utterance, the device comprising:
a network interface;
at least one processor operatively connected to the network interface; and
at least one memory operatively connected to the at least one processor,
wherein the at least one memory stores an instruction that, when executed, causes the at least one processor to:
receive first data associated with a user input obtained through a microphone, from a first external electronic apparatus including the microphone through the network interface, wherein the user input includes a request for performing a first task using a second external electronic apparatus and a second task dependent on the first task using a third external electronic apparatus,
determine a first sequence of states for the second external electronic apparatus to perform the first task and a second sequence of states for the third external electronic apparatus to perform the second task, and

transmit a response including information about the first sequence of states and the second sequence of states to the first external electronic apparatus or an external server configured to control the second external electronic apparatus and the third external electronic apparatus, wherein the information includes a first command for having the first sequence of states for the second external electronic apparatus to perform the first task and a second command for having the second sequence of states for the third external electronic apparatus to perform the second task.

10. The device of claim 9 ,
wherein the first command includes the first sequence of states, and
wherein the second command includes the second sequence of states.

11. The device of claim 9 , wherein the information includes a data packet including a header and a field indicating a plurality of commands.

12. The device of claim 9 , wherein the first sequence of states and the second sequence of states are provided in a JavaScript object notation (JSON) format.

13. The device of claim 9 , wherein the information includes a relationship between the first command and the second command.

14. The device of claim 9 , wherein the first command includes a first device type associated with the second external electronic apparatus and a second device type associated with the third external electronic apparatus.

15. The device of claim 9 , wherein the at least one memory stores an instruction that, when executed, further causes the at least one processor to:
when the first external electronic apparatus and one of the second external electronic apparatus or the third external electronic apparatus are not included in a local network, transmit the information to the external server.

16. The device of claim 9 ,
wherein the first command allows the second external electronic apparatus to have the second sequence of states to generate and transmit second data to the third external electronic apparatus, and
wherein the third external electronic apparatus performs an operation based on the second sequence of states using the second data.

17. The device of claim 16 , wherein, when the second external electronic apparatus and the third external electronic apparatus are in

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Joo Hwan
• Lee Sun Key
• Kim Kyung Tae
• Yeo Jae Yung
• Woo Kyoung Gu
• Yun Jeong Hyeon
• Lim Dong Sun
• Je Seong Min

Classifications

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• H04L67/60

  Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70) · CPC title

• H04L69/22

  Parsing or analysis of headers · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title