Motion stabilization and detection of articulated objects

Patent metadata

Field	Value
Publication number	US-9734404-B2
Application number	US-201514622364-A
Country	US
Kind code	B2
Filing date	Feb 13, 2015
Priority date	May 16, 2013
Publication date	Aug 15, 2017
Grant date	Aug 15, 2017

Abstract

The techniques and systems described herein are directed to isolating part-centric motion in a visual scene and stabilizing (e.g., removing) motion in the visual scene that is associated with camera-centric motion and/or object-centric motion. By removing the motion that is associated with the camera-centric motion and/or the object-centric motion, the techniques are able to focus motion feature extraction mechanisms (e.g., temporal differencing) on the isolated part-centric motion. The extracted motion features may then be used to recognize and/or detect the particular type of object and/or estimate a pose or position of a particular type of object.

First claim

What is claimed is:

1. A system comprising:
one or more image capture devices that capture a sequence of images;
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the system to:
compute optical flow of movement of an object within the sequence of images;
extract motion features based at least in part on the optical flow of the movement of the object within the sequence of images, wherein the motion features characterize part-centric motion that is associated with typical movement of an individual part of a particular type of object and that is useful in distinguishing the particular type of object from other types of objects; and
determine, based at least in part on the motion features, that the optical flow of the movement of the object characterizes the particular type of object within the sequence of images.

2. The system as recited in claim 1 , wherein the instructions further cause the system to compute the optical flow of the movement of the object based on individual groups of pixels within the sequence of images.

3. The system as recited in claim 2 , wherein the optical flow of the movement of the object based on the individual groups of pixels is represented as at least one coarse optical flow field.

4. The system as recited in claim 3 , wherein the instructions further cause the system to determine, based at least in part on the at least one coarse optical flow field, that one or more regions within the sequence of images contain at least a threshold amount of motion, wherein the motion features are extracted from the one or more regions within the sequence of images that contain at least the threshold amount of motion and not from one or more other regions within the sequence of images that do not contain at least the threshold amount of motion.

5. The system as recited in claim 4 , wherein the instructions further cause the system to apply one or more detection windows to the one or more regions within the sequence of images that contain at least the threshold amount of motion, wherein a respective size of an individual detection window is based at least in part on a distance between the one or more image capture devices and the object.

6. The system as recited in claim 1 , wherein the instructions further cause the system to determine that the sequence of images is captured during a period of time that meets or exceeds a predetermined minimum period of time in which it typically takes the individual part of the particular type of object to complete at least one periodic movement, and wherein the optical flow of the movement of the object is computed based at least in part on the determination that the sequence of images is captured during the period of time that meets or exceeds the predetermined minimum period of time.

7. The system as recited in claim 1 , wherein the instructions further cause the system to determine that a number of images in the sequence of images meets or exceeds a predetermined minimum number of images, and wherein the optical flow of the movement of the object is computed based at least in part on the determination that the number of images in the sequence of images meets or exceeds the predetermined minimum number of images.

8. The system as recited in claim 1 , wherein the instructions further cause the system to output a warning in response to determining that the optical flow of the movement of the object characterizes the particular type of object within the sequence of images.

9. The system as recited in claim 1 , wherein the instructions further cause the system to use one or more trained classification algorithms to determine that the optical flow of the movement of the object characterizes the particular type of object within the sequence of images.

10. The system as recited in claim 1 , wherein the instructions further cause the system to determine that the optical flow of the movement of the object characterizes a specific pose of the particular type of object.

11. A device comprising:
one or more processors;
one or more computer-readable storage media storing computer-executable instructions that, when executed by the one or more processors, cause the device to:
receive a sequence of images from one or more image capture devices;
determine one or more regions within the sequence of images that include part-centric motion of an object by computing optical flow of movement of the object;
identify a typical movement of the object within the one or more regions within the sequence of images based on the part-centric motion of the object; and
associate the typical movement of the object with a particular object type.

12. The device as recited in claim 11 , wherein the computer-executable instructions further case the device to determine one or more other regions within the sequence of images that include motion other than the part-centric motion.

13. The device as recited in claim 12 , wherein the motion other than the part-centric motion comprises at least one of camera-centric motion or object-centric motion.

14. The device as recited in claim 11 , wherein the computer-executable instructions further case the device to use one or more trained classification algorithms to associate the typical movement of the object with the particular object type.

15. The device as recited in claim 14 , wherein the object comprises a human and the part-centric motion is associated with at least one of an arm or a leg.

16. A method comprising:
receiving a sequence of images from one or more image capture devices;
computing, by one or more computing devices, optical flow of movement of an object within the sequence of images;
extract motion features based at least in part on the optical flow of the movement of the object within the sequence of images, wherein the motion features characterize part-centric motion that is associated with typical movement of an individual part of a particular type of object and that is useful in distinguishing the particular type of object from other types of objects; and
determining, based at least in part on the motion features, that the optical flow of the movement of the object characterizes the particular type of object within the sequence of images.

17. The method as recited in claim 16 , further comprising computing the optical flow of the movement of the object based on individual groups of pixels within the sequence of images.

18. The method as recited in claim 17 , wherein the optical flow of the movement of the object based on the individual groups of pixels is represented as at least one coarse optical flow field, the method further comprising:
determining, based at least in part on the at least one coarse optical flow field, that one or more regions within the sequence of images contain at least a threshold amount of motion; and
determining, based at least in part on the at least one coarse optical flow field, one or more other regions within the sequence of images that do not contain at least the threshold amount of motion, wherein the motion features are extracted from the one or more regions within the sequence of images that contain at least the threshold amount of motion and not from the one or more other regions within the sequence of images that do not contain at least the threshold amount of motion.

19. The method as recited in claim 18 , further comprising applying one or more detection windows to the one or more regions within the sequence of images that contain at least

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Dollar Piotr
• Zitnick Iii Charles Lawrence
• Park Dennis I

Classifications

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06K9/00711 Primary

  Physics · mapped topic

• G06V40/23 Primary

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06V20/40 Primary

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title