Optimizations for dynamic object instance detection, segmentation, and structure mapping

Patent metadata

Field	Value
Publication number	US-10586350-B2
Application number	US-201815972035-A
Country	US
Kind code	B2
Filing date	May 4, 2018
Priority date	Dec 3, 2017
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

In one embodiment, a system accesses pose probability models for predetermined parts of a body depicted in an image. Each of the pose probability models is configured for determining a probability of the associated predetermined body part being at a location in the image. The system determines a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined body parts. The system further determines a first probability score for the candidate pose based on the pose probability models and the set of coordinates of the candidate pose. A pose representation is generated for the candidate pose using a transformation model and the candidate pose. The system determines a second probability score for the pose representation based on a pose-representation probability model. The system selects the candidate pose to represent a pose of the body based on at least the first and second probability scores.

First claim

What is claimed is:

1. A method comprising, by a computing system:
accessing a plurality of pose probability models for a plurality of predetermined parts of a body that is depicted in an image, respectively, wherein each of the plurality of pose probability models is configured for determining a probability of the associated predetermined part of the body being at a location in the image, wherein the plurality of pose probability models is generated by a machine-learning model;
determining a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined parts of the body in the image;
determining a first probability score for the candidate pose based on the plurality of pose probability models and the set of coordinates of the candidate pose;
generating a pose representation for the candidate pose using a transformation model and the candidate pose;
determining a second probability score for the pose representation based on a pose-representation probability model; and
selecting the candidate pose to represent a pose of the body depicted in the image based on at least the first probability score and the second probability score.

2. The method of claim 1 ,
wherein each coordinate in the set of coordinates of the candidate pose is defined in a first coordinate system of the image;
wherein the pose representation is defined in a first spatial dimension and is generated by applying the transformation model to a set of normalized coordinates that correspond to the set of coordinates of the candidate pose, respectively; and
wherein each coordinate in the set of normalized coordinates is defined in a second coordinate system that is different from the first coordinate system.

3. The method of claim 2 , further comprising:
reprojecting the pose representation from the first spatial dimension into a second spatial dimension associated with the second coordinate system; and
computing a reprojection error based on the reprojected pose representation and the normalized coordinates;
wherein the selection of the candidate pose is further based on the reprojection error.

4. The method of claim 2 , wherein the pose representation is generated based on differences between the set of normalized coordinates and an aggregate representation of a plurality of sets of normalized coordinates that are associated with a plurality of poses, respectively.

5. The method of claim 2 , wherein the second coordinate system is defined relative to one or more of the predetermined parts of the body.

6. The method of claim 1 , wherein each of the plurality of probability models is a probability heat map.

7. The method of claim 1 , wherein the transformation model is generated using principal component analysis.

8. The method of claim 1 , wherein at least one of the plurality of predetermined parts of the body corresponds to a joint of the body.

9. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
accessing a plurality of pose probability models for a plurality of predetermined parts of a body that is depicted in an image, respectively, wherein each of the plurality of pose probability models is configured for determining a probability of the associated predetermined part of the body being at a location in the image, wherein the plurality of pose probability models is generated by a machine-learning model;
determining a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined parts of the body in the image;
determining a first probability score for the candidate pose based on the plurality of pose probability models and the set of coordinates of the candidate pose;
generating a pose representation for the candidate pose using a transformation model and the candidate pose;
determining a second probability score for the pose representation based on a pose-representation probability model; and
selecting the candidate pose to represent a pose of the body depicted in the image based on at least the first probability score and the second probability score.

10. The system of claim 9 ,
wherein each coordinate in the set of coordinates of the candidate pose is defined in a first coordinate system of the image;
wherein the pose representation is defined in a first spatial dimension and is generated by applying the transformation model to a set of normalized coordinates that correspond to the set of coordinates of the candidate pose, respectively; and
wherein each coordinate in the set of normalized coordinates is defined in a second coordinate system that is different from the first coordinate system.

11. The system of claim 10 , wherein the processors are further operable when executing the instructions to perform operations comprising:
reprojecting the pose representation from the first spatial dimension into a second spatial dimension associated with the second coordinate system; and
computing a reprojection error based on the reprojected pose representation and the normalized coordinates;
wherein the selection of the candidate pose is further based on the reprojection error.

12. The system of claim 10 , wherein the pose representation is generated based on differences between the set of normalized coordinates and an aggregate representation of a plurality of sets of normalized coordinates that are associated with a plurality of poses, respectively.

13. The system of claim 10 , wherein the second coordinate system is defined relative to one or more of the predetermined parts of the body.

14. The system of claim 9 , wherein each of the plurality of probability models is a probability heat map.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:
accessing a plurality of pose probability models for a plurality of predetermined parts of a body that is depicted in an image, respectively, wherein each of the plurality of pose probability models is configured for determining a probability of the associated predetermined part of the body being at a location in the image, wherein the plurality of pose probability models is generated by a machine-learning model;
determining a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined parts of the body in the image;
determining a first probability score for the candidate pose based on the plurality of pose probability models and the set of coordinates of the candidate pose;
generating a pose representation for the candidate pose using a transformation model and the candidate pose;
determining a second probability score for the pose representation based on a pose-representation probability model; and
selecting the candidate pose to represent a pose of the body depicted in the image based on at least the first probability score and the second probability score.

16. The media of claim 15 ,
wherein each coordinate in the set of coordinates of the candidate pose is defined in a first coordinate system of the image;
wherein the pose representation is defined in a first spatial dimension and is generated by applying the transformation model to a set of normalized coordinates that correspond to the set of coordi

Assignees

• Facebook Inc

Inventors

• Vajda Peter
• Zhang Peizhao
• Yang Fei
• Wang Yanghan

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/75 Primary

  involving models · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title