Electronic device and operating method thereof for outputting response to user input, by using application

Patent metadata

Field	Value
Publication number	US-2020005784-A1
Application number	US-201916443014-A
Country	US
Kind code	A1
Filing date	Jun 17, 2019
Priority date	Jun 15, 2018
Publication date	Jan 2, 2020
Grant date	—

Abstract

A method of outputting a response to a user input in an electronic device is provided. The method includes receiving a user input from a user and, in response to receiving the user input, generating a first response comprising first content based on the user input, obtaining contextual information of the user, generating a second response comprising second content based on the contextual information, the second content being different from the first content, generating a combined response based on the first response and the second response, and outputting the combined response.

First claim

What is claimed is:

1 . A method of an electronic device, comprising:
receiving a user input from a user; and in response to receiving the user input,
generating a first response comprising first content based on the user input,
obtaining contextual information of the user,
generating a second response comprising second content based on the contextual information, the second content being different from the first content,
generating a combined response based on the first response and the second response, and
outputting the combined response.

2 . The method of claim 1 ,
wherein a type of the first content comprises at least one of text, a moving picture, an image, or audio content, and wherein a type of the second content comprises at least one of text, a moving picture, an image, audio content, a light-emitting diode (LED) output, a vibration output, a visual effect, an audible effect, or a user interface.

3 . The method of claim 1 , further comprising, in response to the user input, obtaining first data,
wherein the first response is generated based on the first data, and wherein the second response is generated based on second data obtained by modifying the first data based on the contextual information.

4 . The method of claim 1 , further comprising inputting the contextual information into a generative model to generate the second response.

5 . The method of claim 4 , further comprising training the generative model based on training data,
wherein the training data comprises information related to the user that was collected by the electronic device, and wherein the generative model is trained based on correlation between the training data and the second response.

6 . The method of claim 1 , wherein the generating of the second response comprises:
obtaining feature information of the first content; and generating the second response comprising the second content based on the feature information and the contextual information.

7 . The method of claim 1 ,
wherein the contextual information comprises at least one feature corresponding to each context of the user, wherein the at least one feature of the contextual information is determined based on feedback information from the user with respect to the combined response, and wherein the contextual information is obtained based on the determined at least one feature.

8 . An electronic device comprising:
a input device configured to receive a user input from a user; at least one processor configured to:
in response to the user input,
generate a first response comprising first content based on the user input,
obtain contextual information of the user,
generate a second response comprising second content based on the contextual information, the second content being different from the first content,
generate a combined response based on the first response and the second response; and

an output device configured to output the combined response.

9 . The electronic device of claim 8 ,
wherein a type of the first content comprises at least one of text, a moving picture, an image, or an audio, and wherein a type of the second content comprises at least one of text, a moving picture, an image, an audio, a light-emitting diode (LED) output, a vibration output, a visual or audible effect, or a user interface.

10 . The electronic device of claim 8 ,
wherein the first response is generated based on first data obtained based on the user input, and wherein the second response is generated based on second data obtained by modifying the first data based on the contextual information.

11 . The electronic device of claim 8 ,
wherein the second response is generated by inputting the contextual information to a generative model.

12 . The electronic device of claim 11 ,
wherein the at least one processor is further configured to obtain training data comprising information related to the user collected by the electronic device as information used to train the generative model, wherein the generative model is trained based on correlation between the training data and the second response.

13 . The electronic device of claim 8 , wherein the at least one processor is further configured to:
obtain feature information of the first content, and generate the second response comprising the second content based on the feature information of the first content and the contextual information.

14 . The electronic device of claim 8 ,
wherein the contextual information comprises at least one feature corresponding to each context of the user, and wherein the at least one feature is determined based on feedback information from the user on the combined response, and the contextual information is obtained based on the at least one feature.

15 . A non-transitory computer-readable storage medium configured to store one or more computer programs including instructions that, when executed by at least one processor, cause the at least one processor to:
receive a user input from a user; and in response to the user input,
generate a first response comprising first content as a response corresponding to the user input,
obtain contextual information of the user,
generate a second response comprising second content different from the first content based on the contextual information, as a response corresponding to the user input,
combine the first response with the second response, and
output a combined response as a response to the user input.

16 . The computer-readable storage medium of claim 15 , further comprising instructions to cause the at least one processor to:
capture an image of the user; detect facial expression information of the user based on the image; and determine an emotion based on the facial expression information, wherein the combined response is output based on the emotion.

17 . The computer-readable storage medium of claim 15 , wherein the instructions to generate the second response further comprise instructions to cause the at least one processor to:
determine whether the second response can be generated using only local resources; and when the second response cannot be generated using only the local resources, transmit the user input and the contextual information to a cloud server; and receive the second response from the cloud server.

18 . The computer-readable storage medium of claim 15 , wherein the instructions to generate the second response further comprise instructions to cause the at least one processor to:
determine a user preference using on a convoluted neural network (CNN); and generate the second response based on the user preference.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Vadackupurath Mani Parameswaranath
• Bhamidipati Sreevatsa Dwaraka
• Vala Vanraj
• Shariff Mohamed Akram Ulla
• Reddy Kachana Raghunatha
• Namitha
• Rampelli Mahender
• Meher Beda Prakash
• Murthy Sujay Srinivasa
• Vatsal Shubham

Classifications

• G06N3/088 Primary

  Non-supervised learning, e.g. competitive learning · CPC title

• G06V10/768

  using context analysis, e.g. recognition aided by known co-occurring patterns · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title