Extraction of user behavior from depth images

Patent metadata

Field	Value
Publication number	US-9959459-B2
Application number	US-201615069558-A
Country	US
Kind code	B2
Filing date	Mar 14, 2016
Priority date	Mar 8, 2013
Publication date	May 1, 2018
Grant date	May 1, 2018

Abstract

Embodiments described herein use depth images to extract user behavior, wherein each depth image specifies that a plurality of pixels correspond to a user. In certain embodiments, one or more average extremity positions of a user, which can also be referred to as average positions of extremity blobs, are extracted from a depth image. An application is then updated based on the average positions of extremity blobs.

First claim

What is claimed is:

1. For use by a computing system including one or more processors that are running first software instructions, a method for extracting user behavior from a depth image that includes a plurality of pixels that correspond to a user that is interacting with the application running on the one or more processors, the method comprising:
identifying a pixel of the depth image that corresponds to an extremity of the user;
identifying an extremity blob that comprises pixels of the depth image that correspond to the user and are within a specified distance of the pixel identified as corresponding to the extremity of the user;
identifying an average position of the extremity blob by determining an average position of the pixels included in the extremity blob; and
using the average position of the extremity blob to provide input to second software instructions running on the one or more processors, wherein the second software instructions may comprise the first software instructions, wherein the input to the second software instructions running on the one or more processors comprises at least one of the following:
a command or action to control or activate a user interface element of the second software instructions;
a command or action to animate an avatar that is being displayed under control of the second software instructions;
an update to a position or movement of an avatar, character or other on-screen object or image that is being displayed under control of the second software instructions; or
an update to tracking of the user performing an exercise or pose so that feedback or points can be provided by the application to the user.

2. The method of claim 1 , wherein the average position of the extremity blob corresponds to one of a left side, a right side, a top, a bottom or a front of the user.

3. The method of claim 1 , further comprising repeating the steps of the method of claim 1 for one or more further extremities of the user so that an average position of an extremity blob is determined for each of a plurality of extremities of the user.

4. The method of claim 3 , wherein each said average position of an extremity blob corresponds to different one of a left side, a right side, a top, a bottom or a front of the user.

5. The method of claim 1 , wherein the identifying a pixel of the depth image that corresponds to an extremity of the user comprises:
checking pixels in a line extending along a side of the depth image to determine if a pixel in the line corresponds to the user;
incrementally moving the line each time there is a determination that no pixel in the line corresponds to the user; and
in response to determining that a pixel in the line corresponds to the user, identifying the pixel in the line as corresponding to an extremity of the user.

6. The method of claim 1 , wherein:
the identifying an extremity blob includes identifying pixels of the depth image that correspond to the user and are within specified blob boundaries; and
the identifying an average position of the extremity blob comprises:
determining an average x value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries; and
determining an average y value of the pixels of the depth image that correspond to the user and are within the specified blob boundaries;

the average x value corresponds to an average horizontal position in the depth image of the extremity blob; and
the average y value corresponds to an average vertical position in the depth image of the extremity blob.

7. The method of claim 1 , further comprising:
using a capture device, located a distance from the user, to obtain the depth image; and
performing a segmentation process to identify the plurality of pixels of the depth image that correspond to the user;
wherein the depth image specifies, for each of the pixels corresponding to the user, a pixel location and a pixel depth; and
wherein the pixel depth, specified for each of the pixels corresponding to the user, is indicative of a distance between the capture device and a portion of the user represented by the pixel.

8. A system that extracts user behavior from a depth image that includes a plurality of pixels that correspond to a user, the system comprising:
one or more storage devices that store depth images; and
one or more processors in communication with the one or more storage devices, wherein the one or more processors are configured to
identify a pixel of a said depth image that corresponds to an extremity of the user;
identify an extremity blob that comprises pixels of the said depth image that correspond to the user and are within a specified distance of the pixel identified as corresponding to the extremity of the user;
identify an average position of the extremity blob by determining an average position of the pixels included in the extremity blob; and
use the average position of the extremity blob to update software instructions running on the one or more processors, wherein the update to the software instructions running on the one or more processors comprises at least one of the following:
a command or action to control or activate a user interface element of the software instructions;
a command or action to animate an avatar that is being displayed under control of the software instructions;
an update to a position or movement of an avatar, character or other on-screen object that is being displayed under control of the software instructions; or
an update to tracking of the user performing an exercise or pose so that feedback or points can be provided by the software instructions to the user.

9. The system of claim 8 , wherein the average position of the extremity blob corresponds to one of a left side, a right side, a top, a bottom or a front of the user.

10. The system of claim 8 , wherein the one or more processors are configured to identify an average position of an extremity blob for one or more further extremities of the user so that an average position of an extremity blob is identified for each of a plurality of extremities of the user; and wherein the one or more processors are configured to use the average position of at least two of the extremity blobs to update the application.

11. The system of claim 10 , wherein each said average position of an extremity blob corresponds to different one of a left side, a right side, a top, a bottom or a front of the user.

12. The system of claim 8 , wherein in order to identify a pixel of a said depth image that corresponds to an extremity, the one or more processors are configured to:
check pixels in a line extending along a side of the depth image to determine if a pixel in the line corresponds to the user;
incrementally move the line each time there is a determination that no pixel in the line corresponds to the user; and
identify the pixel in the line as corresponding to an extremity of the user, in response to a determination that a pixel in the line corresponds to the user.

13. The system of claim 8 , wherein:
in order to identify an extremity blob, the one or more processors are configured to identify pixels of a said depth image that correspond to the user and are within specified blob boundaries; and
in order to identify an average position of the extremity blob, the one or more processors are configured to:
determine an average x value of the pixels of the said depth image that correspond to the user and are within the specified blob boundaries; and
determine an average y value of the pixels of the said depth image that correspond to the user and are within the specified blob b

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Hoof Jonathan
• Ahmad Anis
• Kennett Daniel

Classifications

• G06V40/23

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• A63F2300/6045

  for mapping control signals received from the input arrangement into game commands · CPC title

• A63F2300/1093

  using visible light · CPC title