Locating and orienting device in space

Patent metadata

Field	Value
Publication number	US-9144744-B2
Application number	US-201313914600-A
Country	US
Kind code	B2
Filing date	Jun 10, 2013
Priority date	Jun 10, 2013
Publication date	Sep 29, 2015
Grant date	Sep 29, 2015

Abstract

Example apparatus and methods concern an improved immersive experience for a video gamer that is provided by controlling a game based on the three dimensional location and orientation of a control and display device held by or otherwise associated with the gamer. The location is determined from data comprising a three dimensional position and an orientation of a portion of a player in a three dimensional space associated with a computerized game. The facing and rotation of the device is determined as a function of both the location of the device and the orientation of the device. The orientation may be determined by data from motion sensors in or on the device. Example apparatus and methods control the computerized game based, at least in part, on the position of the device, the facing of the device, and the rotation of the device.

First claim

What is claimed is:

1. A computer-readable storage device storing computer-executable instructions that when executed by a computer control the computer to perform a method for locating and orienting a device in space, the method comprising:
acquiring first electronic data comprising a three dimensional position and an orientation of a portion of a user in a three dimensional space associated with a computerized interactive experience, where the first data is acquired from a first system configured to produce a depth map of the user in the three dimensional space, or where the first data is generated from an infrared interrogation of the three dimensional space or where the first data is generated from a camera detecting visible light associated with the three dimensional space;
determining, as a function of the first data, a position in the three dimensional space of a device held by the user, the device being a controller configured to control the computerized interactive experience or to display information associated with the computerized interactive experience;
acquiring second electronic data comprising an orientation of the device in the three dimensional space, where the second data is generated by a sensor in the device or where the second data is generated by an apparatus on the device, where the sensor is an accelerometer, a gyroscope, or a compass, and where the apparatus is an infrared reflector, an infrared transmitter, a sound reflector, a sound transmitter, a visible light reflector, or a visible light transmitter;
determining a facing of the device and a rotation of the device as a function of the first data or the second data, and
controlling the computerized interactive experience based, at least in part, on the position of the device, the facing of the device, and the rotation of the device.

2. The computer-readable storage device of claim 1 , comprising:
constructing a representation of a skeleton of the user from the depth map, where the skeleton comprises a plurality of joints of the user, where a joint is described by a location, a bend angle, and an orientation, and
determining the position of the device as a function of a relationship between two or more joints in the representation.

3. The computer-readable storage device of claim 1 , comprising:
upon determining that the depth map is missing information concerning a joint that is relevant to determining the position of the device, estimating the missing information based on a selected subset of the first data.

4. The computer-readable storage device of claim 1 , comprising:
acquiring the second data from the device,
computing, from the second data, a three dimensional facing vector that describes a facing axis perpendicular to a display on the device, and
computing, from the second data, a rotation of the device about the facing axis.

5. The computer-readable storage device of claim 1 , comprising:
filtering the first data to produce a filtered and stable location data;
filtering the second data to produce a filtered and stable orientation data; and
computing the position of the device, the facing of the device, and the rotation of the device as a function of the filtered location data or the filtered orientation data,
where filtering the first data to produce the filtered location data includes passing the first data through a Kalman filter, an extended Kalman filter, an unscented Kalman Filter, a high pass filter, a low pass filter, a Bayesian network filter, or a particle filter, and
where filtering the second data to produce the filtered orientation data includes passing the second data through a Kalman filter, a high pass filter, a low pass filter, a Bayesian network filter, or a particle filter.

6. The computer-readable storage device of claim 5 , comprising:
combining information derived from the filtered and stable location data with information derived from the filtered and stable orientation data according to a weighted blending function, where the weights for the weighted blending function are based, at least in part, on a completeness of the first data, a completeness of the second data, an accuracy of the first data, an accuracy of the second data, a timeliness of the first data, or a timeliness of the second data.

7. The computer-readable storage device of claim 1 , comprising:
dynamically reconfiguring a function for determining the position based, at least in part, on a completeness of the first data, an accuracy of the first data, or a timeliness of the first data, and
dynamically reconfiguring a function for determining a facing of the device and a rotation of the device based, at least in part, on a completeness of the first data, a completeness of the second data, an accuracy of the first data, an accuracy of the second data, a timeliness of the first data, or a timeliness of the second data.

8. The computer-readable storage device of claim 1 , comprising:
determining the position of the device, the facing of the device, or the rotation of the device as a function of a location and an orientation of a hand of the player, or
determining the position of the device, the facing of the device, or the rotation of the device as a function of a location and an orientation of the head of the player or as a function of a direction of a gaze of the player.

9. The computer-readable storage device of claim 1 , where the computerized interactive experience is a computerized game or a digital media with which the user interacts.

10. The computer-readable storage device of claim 1 , where controlling the computerized interactive experience includes selectively altering a display of a game scene, selectively altering a display of an avatar associated with the user, providing an alternate view into a media simulation, changing the alternative view in a simulation space as the user moves through real space, or presenting an item in the alternative view in the context of a real space via the media simulation.

11. A computer-readable storage device storing computer-executable instructions that when executed by a computer control the computer to perform a method, the method comprising:
acquiring, from a first system configured to produce a depth map of a player in a three dimensional space associated with a computerized game, first data comprising a three dimensional position and an orientation of a portion of the player in the three dimensional space;
constructing a representation of a skeleton of the player from the depth map, where the representation of the skeleton is a function of locations, bends, and orientations of a plurality of joints of the player;
determining, as a function of a relationship between two or more joints in the representation of the skeleton, a position in the three dimensional game space of a control and display device held by the player;
acquiring, from a sensor in the control and display device or from an apparatus on the control and display device, second data comprising an orientation of the control and display device in the three dimensional game space, the sensor being an accelerometer, a gyroscope, or a compass, the apparatus being an infrared reflector, an infrared transmitter, a sound reflector, a sound transmitter, a visible light reflector, or a visible light transmitter; determining, from the representation of the skeleton or from the second data, a facing of the control and display device and a rotation of the control and display device as a function of the first data or the second data, where the facing is described by a three dimensional facing vector that describes a facing axis perpendicular to a display on the control and display device;
determining, f

Assignees

• Microsoft Corp

Inventors

• Langlois Eric
• Pinto Ed
• Lopez Ruiz Marcelo
• Manion Todd
• Mathe Zsolt

Classifications

• G06F1/1694

  the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer · CPC title

• A63F13/52

  involving aspects of the displayed game scene · CPC title

• A63F13/20

  Input arrangements for video game devices · CPC title

• A63F13/56 Primary

  Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title