Mobile terminal and controlling method thereof

Patent metadata

Field	Value
Publication number	US-10104208-B2
Application number	US-201615218046-A
Country	US
Kind code	B2
Filing date	Jul 24, 2016
Priority date	Dec 29, 2015
Publication date	Oct 16, 2018
Grant date	Oct 16, 2018

Abstract

Disclosed are a mobile terminal for providing an intelligent agent (IA) service and controlling method thereof. The mobile terminal includes a display unit, a camera, a sensing unit comprising a microphone configured to sense a surrounding voice of the mobile terminal, an angle sensor configured sense an angle of the mobile terminal, and a location sensor configured to sense a location of the mobile terminal, a communication unit configured to transmit/receive data with a server configured to store a context information of a user, and a controller, if a trigger signal including a preset voice signal is sensed through the microphone, activating an intelligent agent (IA), the controller, if a preset input signal is sensed in a state that the IA is activated, activating the camera.

First claim

What is claimed is:

1. A mobile terminal, comprising:
a display;
a camera;
a sensing unit comprising a microphone;
a communication unit configured to communicate with a server; and
a controller configured to:
activate an intelligent agent (IA) in response to a trigger signal, the trigger signal comprising a preset voice signal received via the microphone;
activate the camera in response to a preset input signal sensed in a state that the IA has been activated;
sense at least one object from a preview image recognized within a view angle area of the activated camera or from an image captured via the activated camera; and
receive, via the communication unit, context information associated with the sensed at least one object from the server.

2. The mobile terminal of claim 1 , wherein the controller is further configured to:
extract image context information corresponding to the preview image or the captured image;
extract voice context information corresponding to a voice received via the microphone, the voice received within a preset period of time after the trigger signal is sensed; and
provide IA information based on the extracted image context information and voice context information.

3. The mobile terminal of claim 2 , wherein the controller is further configured to:
extract location context information from location information of the mobile terminal; and
provide IA information based on the extracted image context information and location context information.

4. The mobile terminal of claim 2 , wherein:
the image context information corresponds to at least two preview images sensed within the view angle area of the camera; and
the at least two preview images comprise a first preview image and a second preview image.

5. The mobile terminal of claim 4 , wherein the controller is further configured to:
sense the second preview image within a preset period time after sensing the first preview image; and
provide IA information based on the first preview image and the second preview image when a preset period time expires after sensing the second preview image.

6. The mobile terminal of claim 5 , wherein the controller is further configured to:
extract location context information from location information of the mobile terminal; and
provide the IA information based on the location context information, the first preview image, and the second preview image.

7. The mobile terminal of claim 2 , wherein:
the image context information corresponds to at least two capture images sensed within the view angle area of the camera; and
the at least two capture images comprise a first capture image and a second capture image.

8. The mobile terminal of claim 7 , wherein the controller is further configured to:
extract location context information from location information of the mobile terminal; and
provide IA information based on the location context information, the first capture image, and the second capture image.

9. The mobile terminal of claim 2 , wherein the image context information comprises at least one capture image sensed from the preview image within the view angle area of the camera and at least one screen shot image for content displayed via the display.

10. The mobile terminal of claim 9 , wherein context information of a user is stored in the server and the controller is further configured to:
cause the communication unit to receive additional context information of the user sensed by an external device; and
provide IA information based on the received additional context information and the image context information.

11. The mobile terminal of claim 2 , wherein the IA information is provided as at least one of visual information or voice information.

12. The mobile terminal of claim 2 , wherein the IA information comprises tag information indicating a keyword included in the image context information or the voice context information.

13. The mobile terminal of claim 1 , wherein while the IA has been activated, the controller is further configured to:
extract image context information corresponding to at least one screen shot image for content displayed via the display; and
provide IA information based on the extracted image context information.

14. The mobile terminal of claim 13 , wherein the controller is further configured to cause the communication unit to transmit the IA information to the server when an IA of an external device for which user authentication is activated.

15. The mobile terminal of claim 1 , wherein the controller is further configured to:
execute an IA related application when the IA is activated; and
cause the display to display a preview image sensed in a view angle area of the camera via the IA related application when the camera is activated.

16. A method for controlling a mobile terminal, the method comprising:
sensing, via a sensing unit, a trigger signal, the trigger signal comprising a preset voice signal received via a microphone of the mobile terminal;
activating, via a controller, an intelligent agent (IA) in response to the sensed trigger signal;
sensing, via the sensing unit, a preset input signal in a state that the IA has been activated;
activating, via the controller, a camera of the mobile terminal in response to the preset input signal;
sensing, via the controller, at least one object from a preview image recognized within a view angle area of the activated camera or from an image captured via the activated camera; and
receiving, via a communication unit, context information associated with the sensed at least one object from a server.

17. The method of claim 16 , further comprising:
extracting image context information corresponding to the preview image or the captured image;
extracting voice context information corresponding to a voice received via the microphone within a preset period of time after the trigger signal is sensed; and
providing IA information based on the extracted image context information and voice context information.

18. The method of claim 17 , further comprising:
extracting location context information from location information of the mobile terminal; and
providing IA information based on the extracted image context information and location context information.

19. A mobile terminal, comprising:
a display;
a camera;
a sensing unit comprising a microphone;
a communication unit configured to communicate with a server; and
a controller configured to:
activate an intelligent agent (IA) and the camera in response to a trigger signal and a preset input signal, respectively, the trigger signal comprising a preset voice signal received via the microphone;
cause the display to display an indicator indicating the activation of the intelligent agent;
sense at least one object from a preview image recognized within a view angle area of the activated camera or an image captured via the activated camera; and
receive, via the communication unit, context information associated with the sensed at least one object from the server.

Assignees

• Lg Electronics Inc

Inventors

• Baek Bumhyeon
• Jang Hyukjoon

Classifications

• H04N23/60 Primary

  Control of cameras or camera modules · CPC title

• H04N23/632

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title

• H04W4/02 Primary

  Services making use of location information · CPC title

• H04N5/23293

  Electricity · mapped topic

• G10L17/22

  Interactive procedures; Man-machine interfaces · CPC title