Pose tracking pipeline

Patent metadata

Field	Value
Publication number	US-9465980-B2
Application number	US-201414478778-A
Country	US
Kind code	B2
Filing date	Sep 5, 2014
Priority date	Jan 30, 2009
Publication date	Oct 11, 2016
Grant date	Oct 11, 2016

Abstract

A method of tracking a subject includes receiving from a source a depth image of a scene including the subject. The depth image includes a depth for each of a plurality of pixels. The method further includes identifying pixels of the depth image that image the subject and deriving from the identified pixels of the depth image one or more machine readable data structures representing the subject as a model including a plurality of shapes.

First claim

The invention claimed is:

1. A method performed using a system, comprising:
based on at least one image of at least part of a scene obtained from an image capture device, the at least part of the scene including one or more parts of a human body in a first position, separating image data included in the at least one image and corresponding to the one or more parts of the human body from image data included in the at least one image and corresponding to a background of the at least part of the scene;
developing a first model corresponding to the one or more parts of the human body based at least in part on the image data corresponding to the one or more parts of the human body, the first model including one or more skeletal points;
developing a second model based at least in part on data associated with the one or more skeletal points and including shapes corresponding to the one or more parts of the human body;
displaying the second model; and
controlling operation of a program executing on the system based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body, the based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body performed by the system being based at least in part on the at least one image of the at least part of the scene and an additional one or more images.

2. The method of claim 1 , wherein the shapes include a cylinder or a sphere.

3. The method of claim 1 , wherein the one or more skeletal points include a roll joint, each roll joint utilized to track an axial roll angle.

4. The method of claim 1 , wherein the program is a gaming program and the system is a gaming system.

5. The method of claim 1 , wherein the image capture device includes a depth camera, and the at least one image includes at least one depth image.

6. The method of claim 5 , further comprising:
downsampling the at least one depth image to a downsampled at least one depth image having a lower processing resolution than the at least one depth image.

7. The method of claim 5 , wherein separating image data includes distinguishing one or more parts of the human body that are to be tracked from background elements in the at least one depth image.

8. The method of claim 7 , wherein the at least one depth image includes two or more depth frames, and the method further comprises:
deriving a depth history image from the two or more depth frames, wherein a depth value for each pixel of the depth history image is set to a deepest depth value the pixel experiences during the two or more depth frames; and
wherein distinguishing one or more parts of the human body includes identifying moving objects in a foreground of the scene from non-moving background elements using the depth history image.

9. The method of claim 8 , wherein identifying moving objects includes determining whether an object is behind a reference plate that is a history of minimum or maximum depth values observed for the non-moving background elements.

10. The method of claim 1 , wherein the one or more parts of the human body belong to a player of an electronic game holding an object, such that motions of the player and the object are utilized to adjust and/or control parameters of the electronic game.

11. The method of claim 1 , wherein controlling operation of the program includes one or more of ending a game, pausing the game, selecting a level of the game, viewing scores associated with the game, or saving the game.

12. A system comprising:
a capture device configured to obtain one or more images of at least part of a scene;
a logic device;
a data-holding device configured to hold instructions executable by the logic device to:
obtain via the capture device at least one image of the at least part of the scene, the scene including one or more parts of a human body in a first position;
separate image data included in the at least one image and corresponding to the one or more parts of the human body from image data included in the at least one image and corresponding to a background of the at least part of the scene;
develop a first model corresponding to the one or more parts of the human body using the image data corresponding to the one or more parts of the human body, the first model including one or more skeletal points;
develop a second model based at least in part on data associated with the one or more skeletal points and including shapes corresponding to the one or more parts of the human body;
display the second model; and
control operation of a program executing on the system based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body, the based at least in part on using the data associated with the one or more skeletal points of the first model to detect movement of the one or more parts of the human body performed by the system being based at least in part on the at least one image of the at least part of the scene and an additional one or more images.

13. A method performed using a system, comprising:
based at least in part on a first set of images of at least part of a scene including one or more parts of a human body in a first position and obtained from a plurality of image capture devices, with each of the plurality of image capture devices located in a different position relative to the other of the plurality of image capture devices, developing a first model corresponding to the one or more parts of the human body in the first position, the first model including one or more skeletal points;
developing a first display model based at least in part on data associated with the one or more skeletal points of the first model and including shapes corresponding to the one or more parts of the human body;
displaying the first display model;
developing a second model based at least in part on a second set of images of at least part of the scene including the one or more parts of the human body in a second position and obtained from the plurality of image capture devices, the second model including the one or more skeletal points
developing a second display model based at least in part on data associated with the one or more skeletal points of the second model and including shapes corresponding to the one or more parts of the human body;
displaying the second display model;
determining differences between the first model and the second model; and
controlling at least a portion of operation of a program executing on the system based at least in part on the differences between the first model and the second model.

14. The method of claim 13 , wherein the shapes include a cylinder or a sphere.

15. The method of claim 13 , wherein the one or more body parts include a human finger.

16. The method of claim 13 , wherein the one or more skeletal points include one or more joints, the one or more joints include roll joints, each roll joint utilized to track axial roll angles.

17. The method of claim 16 , wherein the first model and the second model further include one or more model parts corresponding to an object held by a player of a gaming program, and wherein motions of the player and the object are utilized to adjust and/or control parameters of the gaming program.

18. The method of claim 16 , wherein controlling operation of the program includes one or more of

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Craig Robert Matthew
• Leyvand Tommer
• Peeper Craig
• Al-Ghosien Momin M
• Bronder Matt
• Williams Oliver
• Geiss Ryan M
• Shotton Jamie Daniel Joseph
• Lee Johnny
• Finocchio Mark

Classifications

• A63F2300/1006

  having additional degrees of freedom · CPC title

• G06F3/033

  Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) · CPC title

• A63F13/428

  involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06K9/00335 Primary

  Physics · mapped topic