Skeletal control of three-dimensional virtual world

Patent metadata

Field	Value
Publication number	US-9489053-B2
Application number	US-201514632071-A
Country	US
Kind code	B2
Filing date	Feb 26, 2015
Priority date	Dec 21, 2010
Publication date	Nov 8, 2016
Grant date	Nov 8, 2016

Abstract

A virtual skeleton includes a plurality of joints and provides a machine readable representation of a human target observed with a three-dimensional depth camera. A relative position of a hand joint of the virtual skeleton is translated as a gestured control, and a three-dimensional virtual world is controlled responsive to the gestured control.

First claim

The invention claimed is:

1. A data holding device holding instructions executable by a logic device, the instructions comprising:
instructions to render a three-dimensional environment for display on a display device;
instructions to receive a machine-readable virtual skeleton structurally representing a human being sighted by a depth camera in a volume large enough to envelop the human being, the virtual skeleton including a hand-joint position in three dimensions corresponding to a position of a hand of the human being;
instructions to render a control cursor for display on the display device, a screen position of the control cursor tracking the hand-joint position;
instructions to lock the control cursor to an object in the three-dimensional environment if a grab threshold of the object is overcome;
instructions to, when the control cursor is locked to the object, move the object in all three dimensions to track the hand-joint position anywhere in the volume, such that movement of the hand effects a corresponding movement of the object in the three-dimensional environment; and
instructions to unlock the control cursor from the object at a release position of the object within the three-dimensional environment if a release threshold of the object is overcome.

2. The data holding device of claim 1 , where the grab threshold of the object is overcome if the hand is closed.

3. The data holding device of claim 1 , where the grab threshold of the object is overcome if the hand is within a threshold distance of the object for a threshold duration.

4. The data holding device of claim 1 , where the grab threshold of the object is overcome if the hand is within a threshold distance of the object and a speed of the hand is below a threshold speed for a threshold duration.

5. The data holding device of claim 1 , where the release threshold is overcome if the hand is open.

6. The data holding device of claim 1 , wherein the instructions further comprise:
instructions to, when the control cursor is locked to the object, rotate the object in the three-dimensional environment responsive to a rotation gesture of the virtual skeleton.

7. The data holding device of claim 1 , wherein the instructions further comprise:
instructions to, when the control cursor is locked to the object, scale the object in the three-dimensional environment responsive to a scale gesture of the virtual skeleton.

8. The data holding device of claim 1 , wherein the object is one of a plurality of objects in the three-dimensional environment selected responsive to a selection gesture of the virtual skeleton, and wherein the instructions further comprise:
instructions to lock the control cursor to the plurality of objects if a grab threshold of the plurality of objects is overcome;
instructions to, when the control cursor is locked to the plurality of objects, move the plurality of objects in all three dimensions to track the hand-joint position, such that movement of the hand effects a corresponding movement of the plurality of objects in the three-dimensional environment; and
instructions to unlock the control cursor from the plurality of objects if a release threshold of the plurality of objects is overcome.

9. The data holding device of claim 8 , wherein the instructions further comprise:
instructions to, when the control cursor is locked to the plurality of objects, rotate the plurality of objects in the three-dimensional environment responsive to a rotation gesture of the virtual skeleton.

10. The data holding device of claim 8 , wherein the instructions further comprise:
instructions to, when the control cursor is locked to the plurality of objects, scale the plurality of objects in the three-dimensional environment responsive to a scale gesture of the virtual skeleton.

11. The data holding device of claim 1 , wherein the instructions further comprise:
instructions to recognize a zoom-in gesture in which right and left hands of the human being are separated; and
instructions to magnify a view of the three-dimensional environment on the display responsive to the zoom-in gesture.

12. The data holding device of claim 1 , wherein the instructions further comprise:
instructions to recognize a zoom-out gesture in which right and left hands of the human being are brought closer; and
instructions to demagnify a view of the three-dimensional environment on the display responsive to the zoom-out gesture.

13. A data holding device holding instructions executable by a logic device, the instructions comprising:
instructions to render a three-dimensional environment for display on a display device;
instructions to receive a machine-readable virtual skeleton structurally representing a human being sighted by a depth camera, the virtual skeleton including a hand-joint position in three dimensions corresponding to a position of a hand of the human being;
instructions to render a control cursor as a human-hand image for display on the display device, a screen position of the control cursor tracking the hand-joint position;
instructions to lock the control cursor to an object in the three-dimensional environment if a grab threshold of the object is overcome, where the control cursor when locked to the object is rendered closed on the object;
instructions to, when the control cursor is locked to the object, move the object in all three dimensions to track the hand-joint position, such that movement of the hand effects a corresponding movement of the object in the three-dimensional environment;
instructions to unlock the control cursor from the object at a release position of the object within the three-dimensional environment if a release threshold of the object is overcome, where the control cursor when unlocked from the object is rendered reopened; and
instructions to, responsive to the screen position of the control cursor reaching a scrolling threshold at a visible edge of the three-dimensional environment, scrolling the three-dimensional environment such that a previously hidden portion of the three-dimensional environment adjacent to the visible edge of the three-dimensional environment becomes unhidden.

14. A data holding device holding instructions executable by a logic device, the instructions comprising:
instructions to render a three-dimensional environment for display on a display device;
instructions to receive a machine-readable model including a plurality of points defined with three-dimensional positional coordinates, the machine-readable model structurally representing a human being sighted by a depth camera, the machine-readable model including a modeled hand position in three dimensions corresponding to an actual position of a hand of the human being;
instructions to render a visually closable control cursor for display on the display device, a screen position of the control cursor tracking the modeled hand position;
instructions to lock the control cursor to an object in the three-dimensional environment if a grab threshold of the object is overcome, where the control cursor when locked to the object is rendered closed;
instructions to, when the control cursor is locked to the object, move the object in all three dimensions to track the modeled hand position, such that movement of the actual position of the hand of the human being effects a corresponding movement of the object in the three-dimensional environment; and
instructions to unlock the control cursor from the object at a release position of the object within the three-dimensional environment if a release threshold of the object is overcome, where the control cursor when unlocked

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Latta Stephen
• Bennett Darren
• Geisner Kevin
• Markovic Relja

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0486

  Drag-and-drop · CPC title

• G06F2203/04806

  Zoom, i.e. interaction techniques or interactors for controlling the zooming operation · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/0485

  Scrolling or panning · CPC title