Gesture recognition areas and sub-areas for interaction with real and virtual objects within augmented reality

Patent metadata

Field	Value
Publication number	US-10466794-B2
Application number	US-201314775776-A
Country	US
Kind code	B2
Filing date	Dec 11, 2013
Priority date	Mar 21, 2013
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

An information processing apparatus includes circuitry that is configured to provide a user interface to control a plurality of controllable objects. The user interface is associated with a gesture recognition area that is defined by a predetermined detection area of a sensor. The gesture recognition area is divided into a plurality of gesture recognition sub-areas based on a number of the controllable objects to be controlled by the user interface. The circuitry is further configured to detect a predetermined gesture within the predetermined detection area, determine the gesture recognition sub-area in which the predetermined gesture is detected, and perform an operation on one of the controllable objects based on the detected predetermined gesture and the determined gesture recognition sub-area in which the predetermined gesture is detected.

First claim

The invention claimed is:

1. An information processing apparatus, comprising:
circuitry configured to:
control a plurality of controllable objects via a user interface, wherein each of the plurality of controllable objects is one of a real object or a virtual object,
associate the user interface with a gesture recognition area that is defined by a detection area of a sensor,
divide the gesture recognition area into a plurality of gesture recognition sub-areas, based on a detection of the plurality of controllable objects controlled by the user interface and a presence of a reference position corresponding to each of the plurality of controllable objects,
assign a priority to each of the plurality of controllable objects based on each of the plurality of controllable objects is one of the real object or the virtual object, wherein:
a first gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a first controllable object of the plurality of controllable objects and a second gesture recognition sub-area of the plurality of gesture recognition sub-areas is associated with a second controllable object of the plurality of controllable objects, and
a first priority associated with the first controllable object is greater than a second priority associated with the second controllable object,

adjust a first size of the first gesture recognition sub-area and a second size of the second gesture recognition sub-area based on the first priority and the second priority, respectively, such that the first size is larger than the second size,
detect a gesture within the detection area,
determine one of the plurality of gesture recognition sub-areas in which the gesture is detected, and
execute an operation on one of the plurality of controllable objects based on the detected gesture in the determined one of the plurality of gesture recognition sub-areas.

2. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:
assign the one of the plurality of controllable objects to the one of the plurality of gesture recognition sub-areas, wherein the one of the plurality of controllable objects is a remote controllable device, and
execute the operation based on a command transmitted by the information processing apparatus to the remote controllable device.

3. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to allocate, for each of the plurality of gesture recognition sub-areas, a sub-region of the plurality of gesture recognition sub-areas based on a center position of each of the plurality of controllable objects corresponding to each of the plurality of gesture recognition sub-areas.

4. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:
indicate at least one candidate gesture, and
associate a candidate operation for one of the plurality of controllable objects based on the candidate gesture.

5. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:
display a menu on a display screen of the information processing apparatus, wherein:
the menu includes a plurality of selectable items, and
the menu is associated with the real object of the plurality of controllable objects present in the one of the plurality of gesture recognition sub-areas, and
select one of the plurality of items based on the detected gesture in the one of the plurality of gesture recognition sub-areas assigned to the real object.

6. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to control a display screen to display a visual indication of the one of the plurality of gesture recognition sub-areas based on the detection of the gesture.

7. The information processing apparatus according to claim 6 , wherein the circuitry is further configured to:
control the display screen to display the visual indication of the one of the plurality of gesture recognition sub-areas, and
control the display screen to display a mark on the one of the plurality of controllable objects, wherein the mark on the one of the plurality of controllable objects indicates that the one of the plurality of controllable objects is a target of the operation on the display screen.

8. The information processing apparatus according to claim 1 , further comprising:
a display screen implemented in a head-mounted display unit, wherein the display screen is configured to present the user interface.

9. The information processing apparatus according to claim 1 , wherein:
the plurality of controllable objects includes the virtual object assigned to a third gesture recognition sub-area of the plurality of gesture recognition sub-areas, and the real object assigned to a fourth gesture recognition sub-area of the plurality of gesture recognition sub-areas, and
the circuitry is further configured to control the virtual object and the real object via the user interface, wherein the real object is recognized based on an image captured by the sensor.

10. The information processing apparatus according to claim 1 , wherein the sensor is a camera.

11. The information processing apparatus according to claim 1 , wherein:
the plurality of controllable objects includes the real object, and
the circuitry is further configured to set one of the plurality of gesture recognition sub-areas based on a recognition of a location of the real object.

12. The information processing apparatus according to claim 11 , wherein the circuitry is further configured to superimpose the user interface over a user view of the real object.

13. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to set a boundary of each of the plurality of gesture recognition sub-areas based on relative locations of the plurality of controllable objects associated with each of the plurality of gesture recognition sub-areas.

14. The information processing apparatus according to claim 1 ,
wherein the circuitry is further configured to set the one of the plurality of gesture recognition sub-areas with a priority based on the real object of the plurality of controllable objects is in one of the plurality of gesture recognition sub-areas.

15. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to:
detect a plurality of gestures within the detection area;
merge each of the plurality of gesture recognition sub-areas based on the detection of a first gesture of the plurality of gestures; and
remove at least one of the plurality of gesture recognition sub-areas based on the detection of a second gesture of the plurality of gestures.

16. The information processing apparatus according to claim 1 , wherein the circuitry is further configured to determine the reference position of each of the plurality of controllable objects based on a position of each of the plurality of controllable objects within a display screen.

17. A method, comprising:
controlling, by circuitry of an information processing apparatus, a plurality of controllable objects via a user interface, wherein each of the plurality of controllable objects is one of a real object or a virtual object,
associating the user interface with a gesture recognition area that is defined by a detection area of a sensor,
dividing the gesture recognition area into a plurality of gesture recognition sub-areas, based on a detection of the

Assignees

• Sony Corp

Inventors

• Maeda Yoshinori
• Yokono Jun
• Minamino Katsuki

Classifications

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title