Apparatus and method for automatic action selection based on image context

Patent metadata

Field	Value
Publication number	US-9436887-B2
Application number	US-201314137373-A
Country	US
Kind code	B2
Filing date	Dec 20, 2013
Priority date	Mar 15, 2013
Publication date	Sep 6, 2016
Grant date	Sep 6, 2016

Abstract

Devices and a method are provided for providing context-related feedback to a user. In one implementation, the method comprises capturing real time image data from an environment of the user. The method further comprises identifying in the image data a hand-related trigger. Multiple context-based alternative actions are associated with the hand-related trigger. Further, the method comprises identifying in the image data an object associated with the hand-related trigger. The object is further associated with a particular context. Also, the method comprises selecting one of the multiple alternative actions based on the particular context. The method further comprises outputting the context-related feedback based on a result of the executed alternative action.

First claim

What is claimed is:

1. An apparatus for providing context-related feedback to a user, the apparatus comprising:
an image sensor configured to capture real time image data from an environment of the user;
at least one processor device configured to:
identify in the image data a hand-related trigger, wherein multiple context-based alternative actions are associated with the hand-related trigger;
identify in the image data an object based on the hand-related trigger, wherein the object is associated with a particular context;
select one of the multiple alternative actions associated with the hand-related trigger based on the particular context associated with the identified object,
execute the selected alternative action; and
output the context-related feedback based on a result of the executed alternative action.

2. The apparatus of claim 1 , wherein the hand-related trigger includes an identification in the image data of at least a portion of the user's hand.

3. The apparatus of claim 1 , wherein the at least one processor device is further configured to identify a plurality of hand-related triggers, and each hand-related trigger is associated with different multiple context-based alternative actions.

4. The apparatus of claim 1 , wherein the hand-related trigger includes an erratic movement of the object.

5. The apparatus of claim 1 , wherein the hand-related trigger includes an atypical movement of the object.

6. The apparatus of claim 1 , wherein the at least one processor device is further configured to identify in the image data a persistence of the hand-related trigger; select a differing alternative action from the multiple alternative actions; and execute the selected differing alternative action after outputting the context-related feedback.

7. The apparatus of claim 1 , wherein the image sensor s configured to be movable with a head of the user.

8. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine the particular context using an identity of the object.

9. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine the particular context using a background of the object.

10. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine the particular context using a location of the hand-related trigger in at least one frame of the image.

11. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine the particular context using information indicative of a location of the user.

12. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine the particular context using information indicative of a time in which the hand-related trigger is identified.

13. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine the particular context using information from at least one previous result of an action executed by the at least one processor.

14. The apparatus of claim 1 , wherein the at least one processor device is further configured to determine movement of a head of the user and to use the head movement determination to identify the object.

15. The apparatus of claim 1 , wherein the alternative action is selected from a group comprising: announcing an identity of an inanimate object, announcing an identity of an individual, scene identification, summing money, monitoring a status of a traffic light, saving an individual's name, audibly reading text, audibly reading text summary, monitoring an objected expected to change, identifying a bus number, identifying currency, identifying a credit card, and identifying a pill.

16. An apparatus for providing context-related feedback to a user, the apparatus comprising:
an image sensor configured to be worn by the user and to capture real time image data from an environment of the user;
at least one processor device configured to:
identify a trigger in the image data, wherein multiple context-based alternative actions are associated with the trigger;
identify an object in the image data based on the trigger;
identify a background of the object in the image data;
determine a particular context using information about at least one of the identified object and the identified background;
select one of the multiple alternative actions associated with the trigger based on the particular context associated with the identified object;
execute the selected alternative action; and
output the context-related feedback based on a result of the executed alternative action.

17. The apparatus of claim 16 , wherein the trigger in the image data includes a hand-related trigger.

18. The apparatus of claim 16 , wherein the trigger in the image data includes an appearance of a new object within a field of view of the image sensor, and wherein the at least one processor device is further configured to:
determine whether the user has focused on the new object.

19. The apparatus of claim 16 , wherein the at least one processor device is further configured to identify a plurality of triggers, and each trigger is associated with a differing alternative action.

20. The apparatus of claim 16 , wherein the at least one processor device is further configured to determine the particular context using information indicative of a location of the user.

21. The apparatus of claim 16 , wherein the at least one processor device is further configured to determine the particular context using information retrieved from audio processing.

22. The apparatus of claim 16 , wherein the at least one processor device is further configured to determine movement of a head of the user and to use the head movement determination to determine the particular context.

23. A method for providing context-related feedback to a user, the method comprising:
capturing real time image data from an environment of the user;
identifying in the image data a hand-related trigger, wherein multiple context-based alternative actions are associated with the hand-related trigger;
identifying in the image data an object based on the hand-related trigger, wherein the object is further-associated with a particular context;
selecting one of the multiple alternative actions associated with the hand-related trigger based on the particular context associated with the identified object; and
outputting the context-related feedback based on a result of the executed alternative action.

24. A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out the method of claim 23 .

Assignees

• Wexler Yonatan
• Na'Aman Erez
• Shashua Amnon
• Orcam Technologies Ltd

Inventors

• Wexler Yonatan
• Na'Aman Erez
• Shashua Amnon

Classifications

• G06V10/243

  by compensating for image skew or non-uniform image deformations · CPC title

• G09B21/006 Primary

  using audible presentation of the information · CPC title

• H04N23/51

  Housings · CPC title

• H04N23/50

  Constructional details · CPC title

• H04N23/661

  Transmitting camera control signals through networks, e.g. control via the Internet · CPC title