Human action data set generation in a machine learning system

Patent metadata

Field	Value
Publication number	US-10679044-B2
Application number	US-201815934315-A
Country	US
Kind code	B2
Filing date	Mar 23, 2018
Priority date	Mar 23, 2018
Publication date	Jun 9, 2020
Grant date	Jun 9, 2020

Abstract

Methods, apparatuses, and computer-readable mediums for generating human action data sets are disclosed by the present disclosure. In an aspect, an apparatus may receive a set of reference images, where each of the images within the set of reference images includes a person, and a background image. The apparatus may identify body parts of the person from the set of reference image and generate a transformed skeleton image by mapping each of the body parts of the person to corresponding skeleton parts of a target skeleton. The apparatus may generate a mask of the transformed skeleton image. The apparatus may generate, using machine learning, a frame of the person formed according to the target skeleton within the background image.

First claim

What is claimed is:

1. A computer-implemented method for generating human action data sets, comprising:
receiving a set of reference images, where each image within the set of reference images includes a person;
identifying body parts of the person in the set of reference images;
generating a transformed skeleton image by mapping each of the body parts of the person to corresponding skeleton parts of a target skeleton, wherein a body position of the target skeleton is different from a body position of the person in the set of reference images;
generating a mask of the transformed skeleton image;
receiving a background image; and
generating, using a machine learning frame generator, a frame of the person formed according to the target skeleton within the background image, the machine learning frame generator being trained based on the target skeleton, the mask of the transformed skeleton image, and the background image.

2. The computer-implemented method of claim 1 , further comprising:
outputting, using a machine learning frame discriminator, a label indicating whether the frame appears to be a real image, the machine learning frame discriminator being trained to distinguish between real images and fake images received from the machine learning frame generator.

3. The computer-implemented method of claim 1 , further comprising:
receiving an action label; and
generating, using a machine learning skeleton generator, the target skeleton according to the action label, the machine learning skeleton generator being trained to predict a distribution of skeleton sequences conditioned on the action label.

4. The computer-implemented method of claim 3 , further comprising:
outputting, using a machine learning skeleton discriminator, a label indicating whether the target skeleton is human looking, the machine learning skeleton discriminator being trained to distinguish between real images and fake images received from the machine learning frame generator.

5. The computer-implemented method of claim 1 , wherein the target skeleton is one of a plurality of target skeletons indicating sequential motions associated with an action label.

6. The computer-implemented method of claim 1 , wherein the mask of the transformed skeleton image is a mask of a region within the set of reference images where the person was located.

7. The computer-implemented method of claim 1 , further comprising:
adding the frame to a data set stored in a memory.

8. A computing device for generating human action data sets, comprising:
a memory;
at least one processor coupled to the memory, wherein the at least one processor is configured to:
receive a set of reference images, where each image within the set of reference images includes a person;
identify body parts of the person in the set of reference images;
generate a transformed skeleton image by mapping each of the body parts of the person to corresponding skeleton parts of a target skeleton, wherein a body position of the target skeleton is different from a body position of the person in the set of reference images;
generate a mask of the transformed skeleton image;
receive a background image; and
generate, using a machine learning frame generator, a frame of the person formed according to the target skeleton within the background image, the machine learning frame generator being trained based on the target skeleton, the mask of the transformed skeleton image, and the background image.

9. The computing device of claim 8 , wherein the at least one processor is further configured to:
output, using a machine learning frame discriminator, a label indicating whether the frame appears to be a real image, the machine learning frame discriminator being trained to distinguish between real images and fake images received from the machine learning frame generator.

10. The computing device of claim 8 , wherein the at least one processor is further configured to:
receive an action label; and
generate, using a machine learning skeleton generator, the target skeleton according to the action label, the machine learning skeleton generator being trained to predict a distribution of skeleton sequences conditioned on the action label.

11. The computing device of claim 10 , wherein the at least one processor is further configured to:
output, using a machine learning skeleton discriminator, a label indicating whether the target skeleton is human looking, the machine learning skeleton discriminator being trained to distinguish between real images and fake images received from the machine learning frame generator.

12. The computing device of claim 8 , wherein the target skeleton is one of a plurality of target skeletons indicating sequential motions associated with an action label.

13. The computing device of claim 8 , wherein the mask of the transformed skeleton image is a mask of a region within the set of reference images where the person was located.

14. The computing device of claim 8 , wherein the at least one processor is further configured to:
add the frame to a data set stored in the memory.

15. A computer-readable storage device storing code executable by one or more processors for generating human action data sets, the code comprising instructions for:
receiving a set of reference images, where each image within the set of reference images includes a person;
identifying body parts of the person in the set of reference images;
generating a transformed skeleton image by mapping each of the body parts of the person to corresponding skeleton parts of a target skeleton, wherein a body position of the target skeleton is different from a body position of the person in the set of reference images;
generating a mask of the transformed skeleton image;
receiving a background image; and
generating, using a machine learning frame generator, a frame of the person formed according to the target skeleton within the background image, the machine learning frame generator being trained based on the target skeleton, the mask of the transformed skeleton image, and the background image.

16. The computer-readable storage device of claim 15 , further comprising code for:
outputting, using a machine learning frame discriminator, a label indicating whether the frame appears to be a real image, the machine learning frame discriminator being trained to distinguish between real images and fake images received from the machine learning frame generator.

17. The computer-readable storage device of claim 15 , further comprising code for:
receiving an action label; and
generating, using a machine learning skeleton generator, the target skeleton according to the action label, the machine learning skeleton generator being trained to predict a distribution of skeleton sequences conditioned on the action label.

18. The computer-readable storage device of claim 17 , further comprising code for:
outputting, using a machine learning skeleton discriminator, a label indicating whether the target skeleton is human looking, the machine learning skeleton discriminator being trained to distinguish between real images and fake images received from the machine learning frame generator.

19. The computer-readable storage device of claim 15 , wherein the target skeleton is one of a plurality of target skeletons indicating sequential motions associated with an action label.

20. The computer-readable storage device of claim 15 , further comprising code for:
adding the frame to a data set stored in a memory.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Vaezi Joze Hamidreza
• Zharkov Ilya
• Pradeep Vivek
• Khodabandeh Mehran

Classifications

• G06N20/00

  Machine learning · CPC title

• G06K9/00342 Primary

  Physics · mapped topic

• G06T2207/20044

  Skeletonization; Medial axis transform · CPC title

• G06K9/00369

  Physics · mapped topic

• G06T7/248

  involving reference images or patches · CPC title