Extraction of user behavior from depth images

Patent metadata

Field	Value
Publication number	US-9311560-B2
Application number	US-201514824869-A
Country	US
Kind code	B2
Filing date	Aug 12, 2015
Priority date	Mar 8, 2013
Publication date	Apr 12, 2016
Grant date	Apr 12, 2016

Abstract

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, information indicative of an angle and/or curvature of a user's body is extracted from a depth image. This can be accomplished by fitting a curve to a portion of a plurality of pixels (of the depth image) that correspond to the user, and determining the information indicative of the angle and/or curvature of the user's body based on the fitted curve. An application is then updated based on the information indicative of the angle and/or curvature of the user's body. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

First claim

What is claimed is:

1. A system that uses depth images to extract user behavior, comprising:
a communication interface that receives depth images;
one or more storage devices that store depth images;
a display interface; and
one or more processors in communication with the one or more storage devices and the display interface, wherein for each of a plurality of depth images the one or more processors are configured to
determine which pixels of the depth image correspond to a user;
identify a subset of pixels of interest from the pixels that correspond to the user;
fit a curve to the subset of pixels of interest to thereby produce a fitted curve comprising a plurality of straight line segments;
identify a first straight line that extends between endpoints of the fitted curve;
identify a second straight line that extends orthogonally from the first straight line to a point of the fitted curve that is farthest away from the first straight line; and
determine, based on the first and second straight lines, information indicative of a curvature of the user's body;

wherein the one or more processors are also configured to use the information indicative of the curvature of the user's body to update an application running on the system.

2. The system of claim 1 , wherein the one or more processors are configured to:
determine a length of the first straight line that extends between endpoints of the fitted curve;
determine a length of the second straight line that extends orthogonally from the first straight line to the point of the fitted curve that is farthest away from the first straight line; and
determine a ratio of the length of the first straight line to the length of the second straight line;
wherein the ratio comprises the information indicative of the curvature of the user's body.

3. The system of claim 1 , wherein the subset of pixels of interest comprise pixels that correspond to a torso of the user.

4. The system of claim 1 , wherein the subset of pixels of interest comprise pixels that correspond to a torso, legs and head of the user.

5. The system of claim 1 , wherein the subset of pixels of interest comprise pixels that correspond to an upper peripheral portion of the user.

6. The system of claim 1 , wherein the subset of pixels of interest comprise pixels that correspond to a lower peripheral portion of the user.

7. The system of claim 1 , further comprising:
a capture device that obtains the depth images;
wherein the communication interface receives the depth images from the capture device.

8. The system of claim 7 , wherein the capture device includes at least one camera.

9. The system of claim 7 , wherein the capture device includes at least two cameras.

10. The system of claim 7 , wherein the capture devices includes a three-dimensional camera.

11. The system of claim 1 , wherein the one or more processors are configured to use the information indicative of the curvature of the user's body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

12. The system of claim 1 , wherein the one or more processors are configured to use the information indicative of the curvature of the user's body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

13. The system of claim 1 , wherein the one or more processors are configured to use the information indicative of the curvature of the user's body to update a position or pose of an avatar of the user that is displayed using the display interface.

14. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for using depth images to extract user behavior, the method comprising:
receiving a depth image that that include a plurality of pixels that correspond to a user;
identifying a subset of pixels of interest from the pixels that correspond to the user;
fitting a curve to the subset of pixels of interest to thereby produce a fitted curve comprising a plurality of straight line segments;
identifying and determining a length of a first straight line that extends between endpoints of the fitted curve;
identifying and determining a length of a second straight line that extends orthogonally from the first straight line to a point of the fitted curve that is farthest away from the first straight line; and
determine a ratio of the length of the first straight line to the length of the second straight line, wherein the ratio is indicative of a curvature of the user's body; and
using the information indicative of the curvature of the user's body to update an application.

15. The one or more processor readable storage devices of claim 14 , wherein the method further comprises performing a segmentation process to identify the pixels in the depth image that correspond to the user.

16. The one or more processor readable storage devices of claim 14 , wherein the subset of pixels of interest comprise one of the following:
pixels that correspond to a torso of the user;
pixels that correspond to a torso, legs and head of the user;
pixels that correspond to an upper peripheral portion of the user; or
pixels that correspond to a lower peripheral portion of the user.

17. The one or more processor readable storage devices of claim 14 , wherein the method further comprises determining information indicative of an angle of the user's body by determining an angle, relative to a plane, of the first straight line extending between endpoints of the fitted curve, wherein the determined angle is indicative of an angle of the user's body.

18. The one or more processor readable storage devices of claim 14 , wherein the method includes using the information indicative of the curvature of the user's body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

19. The one or more processor readable storage devices of claim 14 , wherein the method include using the information indicative of the curvature of the user's body to determine whether the user is in a position or pose that the user is instructed to position themselves, and to provide feedback about the position or pose of the user.

20. The one or more processor readable storage devices of claim 14 , wherein the method includes using the information indicative of the curvature of the user's body to update a position or pose of an avatar of the user that is displayed.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Hoof Jonathan
• Ahmad Anis
• Kennett Daniel

Classifications

• G06V40/23

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• A63F2300/6045

  for mapping control signals received from the input arrangement into game commands · CPC title

• A63F2300/1093

  using visible light · CPC title