Apparatus and method for providing information by recognizing user's intentions

Patent metadata

Field	Value
Publication number	US-10073519-B2
Application number	US-201414335977-A
Country	US
Kind code	B2
Filing date	Jul 21, 2014
Priority date	Aug 26, 2013
Publication date	Sep 11, 2018
Grant date	Sep 11, 2018

Abstract

A mobile device and operating method provides information by automatically recognizing a user intention. The mobile device includes a sensor configured to sense a user's eye movement, an event detector configured to detect an event by inferring a user intention based on the user's eye movement, an application function executor configured to execute a function of an application in response to the detected event. The mobile device may further include a display configured to display results of function execution.

First claim

What is claimed is:

1. A mobile device, comprising:
plural sensors configured to sense eye movements of the user;
an event detector configured to:
selectively control which of the plural sensors are used to sense the eye movements of the user based on a determined user operation;
recognize cognitive context based on the sensed eye movements;
when the user operation is determined to correspond to object recognition operation, infer whether the user has an object recognition intention, based on the recognized cognitive context using a database storing a predetermined correlation between recognition eye movement and cognitive context for the object recognition operation; and
when the user operation is determined to correspond to language translation operation, infer whether the user has a reading intention, based on the recognized cognitive context using a database storing a predetermined correlation between reading eye movement and cognitive context for the language translation operation;

a determiner configured to determine whether the object recognition intention or the reading intention is inferred;
an application function executor configured to:
execute an object recognition function of an object recognition application, as the determined user operation, when the object recognition intention is determined to be inferred in the inferring; and
execute a translation function of a translation application, as the determined user operation, when the reading intention is determined to be inferred in the inferring; and

a display configured to display a result of the object recognition function execution or of the translation function execution,
wherein the object recognition function enables the object recognition application to recognize an object that a user is viewing, and to provide the user with information on the recognized object.

2. The mobile device of claim 1 , wherein the plural sensors comprise an eye capturing camera or an electrooculogram (EOG) sensor.

3. The mobile device of claim 1 , wherein the display is an eye glass display (EGD).

4. The mobile device of claim 1 , further comprising a display contents generator configured to generate display content by processing the result of the object recognition function execution or of the translation function execution.

5. The mobile device of claim 4 , wherein the generated display content comprise at least one of text, a 2D image, or a 3D image.

6. The mobile device of claim 1 , further comprising an application executor configured to execute at least one of the object recognition application or the translation application based on instructions of the user.

7. The mobile device of claim 1 , wherein the plural sensors are selectively operated to perform the sensing of the eye movements in response to the object recognition application being executed or in response to the translation application being executed.

8. The mobile device of claim 1 , wherein the mobile device is included in a glasses-type wearable device.

9. The mobile device of claim 1 , wherein the display displays age, occupation, last known address, and marital status of the recognized object as a recognized person, the plural sensors operate to perform the sensing of the sensed eye movements only when the object recognition application or the translation application is running, and the event detector infers whether the user has the reading intention or the object recognition intention based on whether the user is viewing a written character or an object, and the display displays a meeting place of the object.

10. An operating method of a mobile device, comprising:
executing applications according to instructions of a user of the mobile device;
sensing, at plural sensors, eye movements of the user based on the execution of the applications;
selectively controlling which of the plural sensors are used to sense the eye movements of the user based on a determined user operation;
recognizing cognitive context based on the sensed eye movements;
when the user operation is determined to correspond to object recognition operation, inferring whether the user has an object recognition intention, based on the recognized cognitive context using a database storing a predetermined correlation between recognition eye movement and cognitive context for the object recognition operation;
when the user operation is determined to correspond to language translation operation, inferring whether the user has a reading intention, based on the recognized cognitive context using a database storing a predetermined correlation between reading eye movement and cognitive context for the language translation operation;
determining whether the object recognition intention or the reading intention is inferred;
executing an object recognition function of an object recognition application of the applications, as the determined user operation, when the object recognition intention is determined to be inferred in the inferring;
executing a translation function of a translation application of the applications, as the determined user operation, when the reading intention is determined to be inferred in the inferring; and
displaying a result of the object recognition function execution or of the translation function execution,
wherein the object recognition function enables the object recognition application to recognize an object that a user is viewing, and to provide the user with information on the recognized object.

11. The method of claim 10 , wherein the displaying comprises displaying results of the object recognition function execution or of the translation function execution on an EGD screen.

12. The method of claim 10 , further comprising generating display contents by processing results of the object recognition function execution or of the translation function execution.

13. The method of claim 12 , wherein the generated display contents comprise at least one of text, a 2D image, or a 3D image.

14. The method of claim 10 , wherein the cognitive context is recognized based on a correlation data between the eye movements and a connective context of the user.

15. The method of claim 10 , wherein the sensing is operated to perform the sensing of the eye movements only in response to the applications being executed.

16. A mobile device, comprising:
plural sensors configured to sense eye movements of the user;
an event detector configured to:
selectively control which of the plural sensors are used to sense the eye movements of the user based on a determined user operation;
recognize cognitive context based on the sensed eye movements;
when the user operation is determined to correspond to object recognition operation, infer whether the user has an object recognition intention, based on the recognized cognitive context using a database, storing a predetermined correlation between recognition eye movement and cognitive context for the object recognition operation; and
when the user operation is determined to correspond to language translation operation, infer whether the user has a reading intention, based on the recognized cognitive context using a database storing a predetermined correlation between reading eye movement and cognitive context for the language translation operation;

a determiner configured to determine whether the object recognition intention or the reading intention is inferred;
an application function executor configured to:
execute an object recognition function of an object recognition application, as the determined user operation, when the object recognition

Assignees

• Samsung Electronics Co Ltd

Inventors

• Mun Min-Young
• Sung Jae-Mo

Classifications

• G02B27/017

  Head mounted · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title