Pinch And Hold Gesture Navigation On A Head-Mounted Display

Patent metadata

Field	Value
Publication number	US-2016306431-A1
Application number	US-201615071127-A
Country	US
Kind code	A1
Filing date	Mar 15, 2016
Priority date	Apr 15, 2015
Publication date	Oct 20, 2016
Grant date	—

Abstract

A system for performing a pinch and hold gesture is described. The system includes a head-mounted display (HMD) and a glove, which is worn by a hand of the user. Each finger segment of the glove includes a sensor for detecting positions of the finger segment when moved by the hand. The system includes a computing device interfaced with the HMD and the glove. The computing device analyzes data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by at least two of the finger segments. Moreover, the computing device generates image data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location in the scene at which the pinch and hold gesture is associated.

First claim

1 . A system including:
a head-mounted display (HMD) configured to be worn by a user; a glove configured to be worn by a hand of the user, the glove having finger segments for each finger of the hand, each finger segment includes a sensor for detecting positions of the finger segment when moved by the hand; one or more cameras for capturing images of the HMD and images of the glove; a computing device connected to the one or more cameras and interfaced with the HMD and the glove, wherein the computing device is configured to analyze data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by at least two of the finger segments, wherein the computing device is configured for generating image data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location in the scene at which the pinch and hold gesture is associated, wherein while the pinch and hold gesture is active, the computing device is configured for coupling movement of the glove with movement of the scene, such that the scene moves with detected movement of the glove.

2 . The system of claim 1 , wherein the computing device is configured to release the coupling between the movement of the glove and the movement of the scene when the pinch and hold gesture is determined to be deactivated.

3 . The system of claim 1 , wherein the computing device is configured to ignore an effect of movement of the HMD on the movement of the scene during a time period in which the pinch and hold gesture is active.

4 . The system of claim 1 , wherein the computing device is configured to determine whether a rotate gesture is performed by the glove while the pinch and hold gesture is active, wherein the computing device is configured to send data for rotating the scene in response to determining that the rotate gesture is performed.

5 . The system of claim 1 , wherein the computing device is configured to determine whether a slide gesture is performed by the glove while the pinch and hold gesture is active, wherein the computing device is configured to send data for translating the scene in response to determining that the slide gesture is performed, the translation of the scene rendering a non-displayed portion of the scene, the portion not rendered on the HMD before the translation motion occurs.

6 . The system of claim 1 , wherein while the pinch and hold gesture is active, the computing device is configured to ignore movement of a head of the user to avoid changing the scene according to the movement of the head.

7 . The system of claim 1 , wherein the computing device is configured to determine a position of each of the finger segments from the images of the glove, wherein the computing device is configured to determine from the positions of the finger segments that a portion of one of the at least two finger segments is in contact with a portion of another one of the at least two finger segments to determine that the pinch and hold gesture is performed by the at least two finger segments.

8 . The system of claim 1 , wherein the computing device is configured to send data for displaying an image of the hand of the user in the scene, wherein the visual cue is generated at the location in the scene at which the image of the hand performs the pinch and hold gesture.

9 . A method for translating a viewable portion of a scene viewed via a head-mounted display (HMD), the method comprising:
capturing images of the HMD configured to be worn by a user in a real space to determine a view direction and changes in the view direction; capturing images of a hand of the user; tracking the hand of the user from the captured images of the hand; activating a pinning of the scene as viewed via the HMD, such that the pinning fixes the scene to a location of the hand being tracked, the pinning being activated upon detecting a pinch and hold gesture performed by the hand; enabling translation of the scene in response to movement of the hand while the pinning is active, the translation enabling a view beyond a current view of the scene; disabling the translation when the pinning is detected to be inactive, the pinning is inactive when the pinch and hold gesture is no longer detected.

10 . The method of claim 9 , further comprising:
tracking the HMD from the captured images of the HMD; and enabling a change in the viewable portion of the scene when the pinning is detected to be inactive and when movement of the HMD is tracked.

11 . The method of claim 9 , further comprising generating a cue within the viewable portion of the scene to indicate that the pinning is activated.

12 . The method of claim 9 , wherein activating the pinning of the scene is executed when the pinch and hold is determined to be performed for greater than a pre-determined amount of time.

13 . The method of claim 9 , further comprising:
receiving gesture data regarding an additional gesture performed by the user while the pinch and hold gesture is performed; and modifying the viewable portion of the scene according to the gesture data.

14 . The method of claim 9 , further comprising:
capturing images of another hand of the user; tracking the other hand of the user from the captured images of the other hand; facilitating rotation of the scene when from tracking the other hand, it is determined that a rotate gesture is performed by the other hand of the user, wherein the rotation of the scene is facilitated while the scene is fixed to the location of the hand being tracked.

15 . The method of claim 9 , wherein the pinch and hold gesture is no longer detected when a thumb of the hand is not is contact with a finger of the hand.

16 . A method for changing a size of a scene viewed via a head-mounted display (HMD), the method comprising:
capturing images of the HMD configured to be worn by a user in a real space to determine a view direction and changes in the view direction; capturing images of hands of the user; determining from the captured images that each hand performs a pinch and hold gesture; fixing the scene to locations in the scene at which the captured images indicate the pinch and hold gestures are being performed; enabling change in a size of the scene in response to relative changes in separation between the hands while the scene is fixed; and disabling change in size of the scene when one of the pinch and hold gestures from one of the hands is detected from the captured images to have been released.

17 . The method of claim 16 , further comprising:
generating data for zooming-in the scene in response to determining that the separation is increasing; and generating data for zooming-out the scene in response to determining that the separation is decreasing.

18 . The method of claim 16 , further comprising:
determining whether the hands move away or towards the HMD; facilitating zooming-in of the scene upon determining that the hands move towards the HMD; and facilitating zooming-out of the scene upon determining that the hands move away from the HMD.

19 . The method of claim 16 , further comprising:
determining whether the hands move away or towards the HMD in a direction substantially parallel to a width of a torso of the user; facilitating zooming-out of the scene upon determining that the hands move towards the HMD in the direction; facilitating zooming-in of the scene upon determining that the hands move away from the HMD in the direction.

Assignees

• Sony Computer Entertainment Inc

Inventors

• Stafford Jeffrey Roger
• Marks Richard L

Classifications

• G02B2027/0185

  Displaying image at variable distance · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06F2203/04806

  Zoom, i.e. interaction techniques or interactors for controlling the zooming operation · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title