System and method for human pose estimation in unconstrained video

Patent metadata

Field	Value
Publication number	US-10509957-B2
Application number	US-201715425477-A
Country	US
Kind code	B2
Filing date	Feb 6, 2017
Priority date	Feb 5, 2016
Publication date	Dec 17, 2019
Grant date	Dec 17, 2019

Abstract

A system and method for estimating a sequence of human poses in an unconstrained video. In the present invention, a unified two stage, tree-based, optimization problem is solved for which an efficient and exact solution exists. While the proposed method finds an exact solution, it does not sacrifice the ability to model the spatial and temporal constraints between body parts in the video frames on the unconstrained video.

First claim

What is claimed is:

1. A method for estimating human poses in an unconstrained video, the method comprising:
receiving, at a computing device comprising hardware components and software programs, an unconstrained video comprising a plurality of consecutive frames including at least one human pose;
generating a plurality of best full body pose hypotheses for each of the plurality of consecutive frames;
extracting a plurality of real body part nodes from each of the plurality of best full body pose hypotheses;
extracting a plurality of real body part nodes from each of the plurality of best full body pose hypotheses in each of the plurality of consecutive frames of the unconstrained video;
generating a real body part hypotheses for each of the plurality of real body part nodes extracted from the plurality of best full body pose hypotheses;
combining one or more pairs of symmetric real body part nodes into a single abstract coupled body part node to generate a plurality of abstract coupled body part nodes for each of the plurality of consecutive frames of the unconstrained video, wherein each of the one or more pairs of symmetric real body part nodes includes a left real body part node of the at least one human pose and a corresponding symmetric right real body part node of the at least one human pose;
generating a plurality of abstract body part hypotheses from the plurality of abstract coupled body part nodes and each of the real body part hypotheses;
generating an optimal tracklet for each of the abstract body part hypotheses; and
estimating a human pose in the unconstrained video based upon the abstract body part tracklets using tree-based optimization.

2. The method of claim 1 , wherein generating a plurality of best full body pose hypotheses for each of the plurality of consecutive frames, further comprising generating a plurality of best full body pose hypotheses using an N-best inference algorithm.

3. The method of claim 1 , wherein the real body part nodes are selected from head, neck, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right foot and left foot.

4. The method of claim 1 , wherein the abstract body part nodes include abstract single body part nodes.

5. The method of claim 4 , wherein the abstract single body part nodes include, head top and head bottom.

6. The method of claim 1 , wherein the abstract coupled body part nodes include one or more of, right shoulder combined with left shoulder, right elbow combined with left elbow, right wrist combined with left wrist, right hip combined with left hip, right knee combined with left knee and right ankle combined with left ankle.

7. The method of claim 1 , wherein generating an optimal tracklet for each of the abstract body part hypotheses further comprises selecting the one tracklet for each of the abstract body parts that maximizes a combined detection score and compatible score weights.

8. A system for estimating human poses in an unconstrained video, the system comprising:
at least one computing device comprising hardware components and software programs for;
receiving an unconstrained video comprising a plurality of consecutive frames including at least one human pose;
generating a plurality of best full body pose hypotheses for each of the plurality of consecutive frames;
extracting a plurality of real body part nodes from each of the plurality of best full body pose hypotheses in each of the plurality of consecutive frames of the unconstrained video;
generating a real body part hypotheses for each of the plurality of real body part nodes extracted from the plurality of best full body pose hypotheses;
combining one or more pairs of symmetric real body part nodes into a single abstract coupled body part node to generate a plurality of abstract coupled body part nodes for each of the plurality of consecutive frames of the unconstrained video, wherein each of the one or more pairs of symmetric real body part nodes includes a left real body part node of the at least one human pose and a corresponding symmetric right real body part node of the at least one human pose;
generating a plurality of abstract body part hypotheses from the plurality of abstract coupled body part nodes and each of the real body part hypotheses;
generating an optimal tracklet for each of the abstract body part hypotheses; and
estimating a human pose in the unconstrained video based upon the abstract body part tracklets using tree-based optimization.

9. The system of claim 8 , wherein the system further includes software programs for generating a plurality of best full body pose hypotheses for each of the plurality of consecutive frames, further comprising generating a plurality of best full body pose hypotheses using an N-best inference algorithm.

10. The system of claim 8 , wherein the real body part nodes are selected from head, neck, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right foot and left foot.

11. The system of claim 8 , wherein the abstract body part nodes include abstract single body part nodes.

12. The system of claim 11 , wherein the abstract single body part nodes include, head top and head bottom.

13. The system of claim 8 , wherein the abstract coupled body part nodes include one or more of, right shoulder combined with left shoulder, right elbow combined with left elbow, right wrist combined with left wrist, right hip combined with left hip, right knee combined with left knee and right ankle combined with left ankle.

14. The system of claim 1 , wherein the system is integrated into a gaming system or a camera.

15. One or more non-transitory computer-readable media having computer-executable instructions for performing a method of running a software program on a computing device, the computing device operating under an operating system, the method including issuing instructions from the software program comprising:
receiving an unconstrained video comprising a plurality of consecutive frames including at least one human pose;
generating a plurality of best full body pose hypotheses for each of the plurality of consecutive frames;
extracting a plurality of real body part nodes from each of the plurality of best full body pose hypotheses in each of the plurality of consecutive frames of the unconstrained video;
generating a real body part hypotheses for each of the plurality of real body part nodes extracted from the plurality of best full body pose hypotheses;
combining one or more pairs of symmetric real body part nodes into a single abstract coupled body part node to generate a plurality of abstract coupled body part nodes for each of the plurality of consecutive frames of the unconstrained video, wherein each of the one or more pairs of symmetric real body part nodes includes a left real body part node of the at least one human pose and a corresponding symmetric right real body part node of the at least one human pose;
generating a plurality of abstract body part hypotheses from the plurality of abstract coupled body part nodes and each of the real body part hypotheses;
generating an optimal tracklet for each of the abstract body part hypotheses; and
estimating a human pose in the unconstrained video based upon the abstract body part tracklets using tree-based optimization.

16. The media of claim 15 , wherein generating a plurality of best full body pose hypotheses for each of the plurality of consecutive frames, further comprising generating a plurality of best full body pose hypotheses using an N-best inference algorith

Assignees

• Univ Central Florida Res Found Inc

Inventors

• Zhang Dong
• Shah Mubarak

Classifications

• G06V10/426

  Graphical representations · CPC title

• G06V10/84

  using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks · CPC title

• G06V40/103 Primary

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title