Automated costume augmentation using shape estimation

Patent metadata

Field	Value
Publication number	US-10885708-B2
Application number	US-201816162160-A
Country	US
Kind code	B2
Filing date	Oct 16, 2018
Priority date	Oct 16, 2018
Publication date	Jan 5, 2021
Grant date	Jan 5, 2021

Abstract

An automated costume augmentation system includes a computing platform having a hardware processor and a system memory storing a software code. The hardware processor executes the software code to provide an image including a posed figure to an artificial neural network (ANN), receive from the ANN a 2D skeleton data including joint positions corresponding to the posed figure, and determine a 3D pose corresponding to the posed figure using an optimization algorithm applied to the skeleton data. The software code further identifies one or more proportion(s) of the posed figure based on the skeleton data, determines bone directions corresponding to the posed figure using another optimization algorithm applied to the 3D pose, parameterizes a costume for the posed figure based on the 3D pose, the proportion(s), and the bone directions, and outputs an enhanced image including the posed figure augmented with the fitted costume for rendering on a display.

First claim

What is claimed is:

1. An automated costume augmentation system comprising a computing platform including a hardware processor and a system memory storing a software code, the hardware processor configured to execute the software code to:
determine a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function;
identify at least one proportion of the posed figure based on the 2D skeleton data;
determine a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions;
parameterize a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and
output an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.

2. The automated costume augmentation system of claim 1 , further comprising a 3D poses library accessible by the software code, wherein the 3D pose corresponding to the posed figure is determined using the first optimization algorithm applied to the 2D skeleton data and a plurality of 3D poses stored in the 3D poses library.

3. The automated costume augmentation system of claim 1 , wherein the at least one proportion of the posed figure includes at least one of a shoulder-width to hip-width ratio or a shoulder-width to average upper body height ratio of the posed figure.

4. The automated costume augmentation system of claim 1 , wherein, after parameterizing the costume for fitting to the posed figure and before outputting the enhanced image, the hardware processor is further configured to execute the software code to:
cover a body portion of the posed figure and an adjacent background portion of the image with a mask, wherein at least a head of the posed figure is not covered by the mask;
inpaint the mask to produce an inpainted mask having a restored background portion of the image; and
overlay the inpainted mask with the costume to produce the enhanced image.

5. The automated costume augmentation system of claim 1 , wherein the image comprises a single monocular image.

6. The automated costume augmentation system of claim 1 , wherein the computing platform is part of a personal communication device remote from an artificial neural network (ANN) generating the 2D skeleton data, the personal communication device further comprising the display and a camera.

7. The automated costume augmentation system of claim 6 , wherein the hardware processor is further configured to execute the software code to obtain the image using the camera.

8. The automated costume augmentation system of claim 6 , wherein the hardware processor is further configured to execute the software code to render the enhanced image on the display.

9. A method for use by an automated costume augmentation system including a computing platform having a hardware processor executing a software code stored in a system memory, the method comprising:
determining a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function;
identifying at least one proportion of the posed figure based on the 2D skeleton data;
determining a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions;
parameterizing a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and
outputting an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.

10. The method of claim 9 , wherein the automated costume augmentation system further comprises a 3D poses library accessible by the software code, wherein determining the 3D pose corresponding to the posed figure comprises applying the first optimization algorithm to the 2D skeleton data and a plurality of 3D poses stored in the 3D poses library.

11. The method of claim 9 , wherein the at least one proportion of the posed figure includes at least one of a shoulder-width to hip-width ratio or a shoulder-width to average upper body height ratio of the posed figure.

12. The method of claim 9 , wherein the method further comprises, after parameterizing the costume for fitting to the posed figure and before outputting the enhanced image:
covering, by the software code executed by the hardware processor, a body portion of the posed figure and an adjacent background portion of the image with a mask, wherein at least a head of the posed figure is not covered by the mask;
inpainting, by the software code executed by the hardware processor, the mask to produce an inpainted mask having a restored background portion of the image; and
overlaying, by the software code executed by the hardware processor, the inpainted mask with the costume to produce the enhanced image.

13. The method of claim 9 , wherein the image comprises a single monocular image.

14. The method of claim 9 , wherein the computing platform is part of a personal communication device remote from an artificial neural network (ANN) generating the 2D skeleton data, the personal communication device further comprising the display and a camera.

15. The method of claim 14 , further comprising obtaining the image using the camera.

16. The method of claim 14 , further comprising rendering the enhanced image on the display.

17. A computer-readable non-transitory medium having stored thereon a software code including instructions, which when executed by a hardware processor, instantiate a method comprising:
determining a three-dimensional (3D) pose corresponding to a two-dimensional (2D) skeleton data including a plurality of joint positions corresponding to a posed figure in an image, using a first optimization algorithm applied to the 2D skeleton data, the first optimization algorithm including a first objective function;
identifying at least one proportion of the posed figure based on the 2D skeleton data;
determining a plurality of bone directions corresponding to the posed figure, using a second optimization algorithm applied to the 3D pose, wherein the second optimization algorithm includes the first objective function summed with a second objective function configured to match the plurality of bone directions;
parameterizing a costume for fitting to the posed figure based on the 3D pose, the at least one proportion, and the plurality of bone directions; and
outputting an enhanced image including the posed figure augmented with the costume fitted to the posed figure for rendering on a display.

18. The computer-readable non-transitory medium of claim 17 , wherein determining the 3D pose corresponding to the posed figure comprises applying the first optimization algorithm to the 2D s

Assignees

• Disney Entpr Inc
• Eth Zuerich

Inventors

• Guay Martin
• Cimen Gökcen
• Maurhofer Christoph
• Ryffel Mattia
• Sumner Robert W

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20044

  Skeletonization; Medial axis transform · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title