Systems and methods of interacting with a virtual grid in a three-dimensional (3D) sensory space

Patent metadata

Field	Value
Publication number	US-11983401-B1
Application number	US-201715832697-A
Country	US
Kind code	B1
Filing date	Dec 5, 2017
Priority date	Jun 4, 2014
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

The technology disclosed relates to selecting a virtual item from a virtual grid in a three-dimensional (3D) sensory space. It also relates to navigating a virtual modality displaying a plurality of virtual items arranged in a grid by and automatically selecting a virtual item in a virtual grid at a terminal end of a control gesture of a control object responsive to a terminal gesture that transitions the control object from one physical arrangement to another. In one implementation, the control object is a hand. In some implementations, physical arrangements of the control object include at least a flat hand with thumb parallel to fingers, closed, half-open, pinched, curled, fisted, mime gun, okay sign, thumbs-up, ILY sign, one-finger point, two-finger point, thumb point, or pinkie point.

First claim

What is claimed is:

1. A method including:
creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;
generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;
providing the user of the head mounted device with the generated live video stream;
identifying, by the head mounted device, a unique real-world marker that is present in the real-world physical environment from (i) the live video stream or (ii) the real-world physical environment;
identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, that corresponds to and provides information about the identified unique real-world marker;
generating, for display, 3D virtual imagery including (i) a virtual grid with a plurality of grid lines, (ii) virtual imagery corresponding to the identified virtual interactive item, and (iii) virtual imagery corresponding to a plurality of virtual items responsive to gestures in the 3D sensory space, wherein each generated virtual item is in visual correspondence with a different set of gridlines and wherein the generated 3D virtual imagery is superimposed, as a free-floating virtual modality in the real-world physical environment allowing the user to simultaneously view both the 3D virtual imagery and the real-world physical environment;
providing the user of the head mounted device with the generated 3D virtual imagery;
detecting, using images provided in the generated live video stream, a gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;
interpreting the detected gesture as selecting one virtual item from the library of virtual interactive items;
generating, for display by the head mounted device, a menu associated with the selected virtual item;
providing the user of the head mounted device with the generated menu; and
modifying a presentation property of the selected virtual item in the virtual grid by augmenting the selected and displayed virtual item with additional graphics.

2. The method of claim 1 , further including superimposing, by the head mounted device, the identified virtual interactive item onto the identified unique real-world marker.

3. The method of claim 1 , wherein
the head mounted device includes one or more projectors that project imagery into the real-world physical environment and wherein the method further includes projecting, by the one or more projectors of the head mounted device, the identified virtual interactive item onto the identified unique real-world marker in the real-world physical environment, such that other individuals in the real-world physical environment can view the virtual interactive item that is projected into the real-world environment.

4. The method of claim 1 , wherein:
the identifying of the virtual interactive item identifies two or more virtual interactive items, from the library of virtual interactive items, that correspond to and provide information about the identified unique real-world marker; and
the identified two or more virtual interactive items are included in the 3D virtual imagery generated for display.

5. The method of claim 1 , wherein
the detected gesture is a scooping gesture in which a representation of a hand of a user appears to start from a position behind a virtual item and then proceed in a motion that appears to scoop up the virtual item from behind.

6. The method of claim 1 , wherein
the identified virtual interactive item is superimposed in place of the identified unique real-world marker, so as to replace the identified unique real-world marker from view of the user.

7. The method of claim 1 wherein
the unique real-world marker comprises at least one of a two- or three-dimensional barcode.

8. The method of claim 1 wherein
the unique real-world marker comprises an image of a real-world item.

9. A method including:
creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;
generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;
providing the user of the head mounted device with the generated live video stream;
identifying, by the head mounted device, a unique real-world marker that is present in the real-world physical environment from (i) the live video stream or (ii) the real-world physical environment;
identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, that corresponds to and provides information about the identified unique real-world marker;
generating, for display, 3D virtual imagery including (i) a virtual grid with a plurality of grid lines, (ii) virtual imagery corresponding to the identified virtual interactive item, and (iii) virtual imagery corresponding to a plurality of virtual items responsive to gestures in the 3D sensory space, wherein each generated virtual item is in visual correspondence with a different set of gridlines and wherein the generated 3D virtual imagery is superimposed, as a free-floating virtual modality in the real-world physical environment allowing the user to simultaneously view both the 3D virtual imagery and the real-world physical environment;
providing the user of the head mounted device with the generated 3D virtual imagery;
detecting, using images provided in the generated live video stream, a first sweep of a control object responsive to a first control gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;
defining an extent of translation along a first axis of a virtual grid in proportion to length of the detected first sweep of the control object;
detecting, using images provided in the generated live video stream, a second sweep of the control object responsive to a second control gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;
defining an extent of translation along a second axis of the virtual grid in proportion to length of the detected second sweep of the control object, wherein the second axis is perpendicular to the first axis;
automatically selecting a virtual item of the generated virtual items in the virtual grid at a terminal end of the detected second sweep; and
modifying a presentation property of the selected virtual item in the virtual grid by augmenting the selected and displayed virtual item with additional graphics.

10. A method including:
creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;
generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;
providing the user of the head mounted device with the generated live video stream;
identifying, by the head mounted device, a unique real-world marker that is present in the real-world physical environment from (i) the live video stream or (ii) the real-world physical environment;
identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, t

Assignees

• Ultrahaptics Ip Two Ltd
• Ultrahaptics Ip Two Ltd United Kingd

Inventors

• Ku Bingxin
• Chen Pohung
• Cohen Isaac L
• Durdik Paul A

Classifications

• G06F3/04883 Primary

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title