Utilizing real world objects for user input

Patent metadata

Field	Value
Publication number	US-10013083-B2
Application number	US-201414263885-A
Country	US
Kind code	B2
Filing date	Apr 28, 2014
Priority date	Apr 28, 2014
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

Methods, systems, apparatuses and computer-readable media for utilizing real world objects to interact with a user interface are presented. The method may comprise a device processing image data to identify an interactive surface area and an interacting object. Subsequently, the device may determine at least two regions of the interactive surface area. In addition, the device may map commands of a user interface to the at least two regions of the interactive surface area. Subsequently, the device may determine a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area. In addition, the device may perform a mapped command of the user interface, wherein the mapped command is determined based on the selected region.

First claim

What is claimed is:

1. A method of a device, comprising:
processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;
determining at least two regions of the interactive surface area;
tracking each of the at least two regions of the interactive surface area;
mapping commands of a user interface to the at least two regions of the interactive surface area, wherein the mapping commands of a user interface to regions of the interactive surface area further comprises:
selecting a number of commands of the user interface from a plurality of user commands based on how many tracked regions comprise the tracked regions of the interactive surface area, wherein a number of commands of the plurality of commands exceeds how many tracked regions comprise the tracked regions; and
correlating each tracked region with a different command based in part on priorities associated with each command, the priorities having been determined based in part on an application currently active on the device;

determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and
performing a mapped command of the user interface, on the device, wherein the mapped command is determined based on the selected region.

2. The method of claim 1 , wherein the processing the image to identify the interactive surface area is based on a pose of the interactive surface area.

3. The method of claim 1 , wherein the interactive surface area is a hand and the interacting object is a thumb attached to the hand.

4. The method of claim 1 , wherein the interactive surface area is a hand and the interacting object is a second hand.

5. The method of claim 1 , wherein the interactive surface area may be one or more objects.

6. The method of claim 1 , wherein the interacting object may be one or more objects.

7. The method of claim 1 , wherein determining the at least two regions of the interactive surface area is based on physical features of the interactive surface area.

8. The method of claim 1 , wherein the proximity comprises direct contact between the interacting object and the interactive surface area.

9. The method of claim 1 , further comprising tracking the interacting object.

10. The method of claim 2 , wherein the processing the image to identify the interactive surface area is further based on a number of objects in the image, a size of an object, a number of regions on the object, visibility of regions on an object, mobility of the object, the interacting object or any combination thereof.

11. The method of claim 1 , wherein the interacting object is detected based on the number of objects in the image, size of an object, visibility of an object, mobility of an object, the interactive surface area or any combination thereof.

12. The method of claim 1 , further comprising:
determining a motion of the interacting object and whether the motion corresponds to a gesture;
determining boundaries of each region of the interactive surface area that allows gestures;
determining a command correlated to the gesture; and
performing the command if the motion was performed within the boundaries.

13. The method of claim 12 , wherein the motion was at least partially performed within the boundaries.

14. The method of claim 1 , wherein the processing the image to identify the interactive surface area comprises identifying a predetermined movement in the image data that identifies the interactive surface area.

15. A method of a device, comprising:
processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;
determining at least two regions of the interactive surface area;
tracking each of the at least two regions of the interactive surface area;
mapping commands of a user interface to the at least two regions of the interactive surface area;
determining visibility of each tracked region of the interactive surface area;
determining priority of commands mapped on a hidden tracked region and on a visible tracked region responsive to the hidden tracked region being hidden;
remapping commands from the hidden tracked region to the visible tracked region based on the priority of commands, wherein remapping the commands comprises comparing a first priority associated with a first command mapped to a hidden tracked region and a second priority associated with a second command mapped to a visible tracked region, and remapping the first command to the second tracked region responsive to the first priority exceeding the second priority;
determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and
performing a mapped command of the user interface, on the device, wherein the mapped command is determined based on the selected region.

16. The method of claim 15 , wherein the priority of commands are based on a command's likelihood of use, number of commands required, organization of commands, and regions previously associated with the command.

17. An apparatus, comprising:
at least one processor configured to:
process image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;
determine at least two regions of the interactive surface area;
track each of the at least two regions of the interactive surface area;
map commands of a user interface to the at least two regions of the interactive surface area, wherein the at least one processor configured to map commands of a user interface to the at least two regions of the interactive surface area further configured to:
select a number of commands of the user interface from a plurality of user commands based on how many tracked regions comprise the tracked regions of the interactive surface area, wherein a number of commands of the plurality of commands exceeds how many tracked regions comprise the tracked regions; and
correlate each tracked region with a different command based in part on priorities associated with each command, the priorities having been determined based in part on an application currently active on the apparatus;

determine a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area;
perform a mapped command, wherein the mapped command is determined based on the selected region; and

a memory coupled to the at least one processor.

18. The apparatus of claim 17 , wherein the at least one processor configured to determine the at least two regions of the interactive surface area is based on physical features of t

Assignees

• Qualcomm Inc

Inventors

• Amariutei Alexandru
• Hildreth Evan Robbert

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F3/0412 Primary

  Digitisers structurally integrated in a display · CPC title