System and method for 3D try-on based on human pose and body shape estimation

Patent metadata

Field	Value
Publication number	US-11941770-B2
Application number	US-202017138213-A
Country	US
Kind code	B2
Filing date	Dec 30, 2020
Priority date	Dec 30, 2020
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

A method and a system for garment try-on. The method includes: capturing a three dimensional (3D) image of a customer; obtaining first 3D pose of the customer in the 3D image; performing a machine learning model on the 3D pose to generate a first skinned multi-person linear model (SMPL) pose; calculating an angle of the whole body rotation of the customer based on the first SMPL pose; when the angle is in a predefined range relative to a front direction of the customer: constructing an SMPL model using the first SMPL pose; and when the angle is out of the predefined range: generating a second SMPL pose using two dimensional (2D) component of the 3D image, and constructing the SMPL model using the second SMPL pose.

First claim

What is claimed is:

1. A method, comprising:
capturing, by a camera, a three dimensional (3D) image of a customer;
obtaining, by a computing device, a first 3D pose of the customer in the 3D image;
generating, by the computing device performing a machine learning model on the first 3D pose, a first skinned multi-person linear model (SMPL) pose, wherein the first SMPL pose comprises a plurality of rotation matrixes representing rotations of the customer's joints and a global rotation matrix representing whole body rotation of the customer;
calculating, by the computing device, an angle of the whole body rotation of the customer based on the global rotation matrix;
when the angle of the whole body rotation is in a predefined range relative to a front direction of the customer: constructing an SMPL model using the first SMPL pose; and
when the angle of the whole body rotation is out of the predefined range: generating a second SMPL pose using two dimensional (2D) component of the 3D image, and constructing the SMPL model using the second SMPL pose.

2. The method of claim 1 , further comprising: dressing a garment model onto the SMPL model, and displaying the SMPL model dressed in the garment model on a display connected to the computing device.

3. The method of claim 2 , further comprising: purchasing a garment in the garment model according to an instruction from the customer.

4. The method of claim 2 , further comprising: rendering the SMPL model using the image of the customer.

5. The method of claim 2 , further comprising: obtaining an SMPL shape from the image of the customer before the step of constructing the SMPL model, and constructing the SMPL model using the SMPL shape and one of the first and second SMPL poses.

6. The method of claim 1 , wherein the machine learning model comprises:
a non-residual block comprising a first fully-connected layer, a first batchnorm layer, and a first rectified linear unit (ReLU) layer;
a first residual block connected to the non-residual block, comprising a second fully-connected layer, a second batchnorm layer, and a second ReLU layer;
a second residual block connected to the first residual block, comprising a third fully-connected layer, a third batchnorm layer, and a third ReLU layer;
a plurality of joint self-attention blocks connected to the second residual block, each of the plurality of the joint self-attention blocks predicting a feature corresponding to one of a plurality of joints of the customer; and
a global self-attention block connected to the second residual block, predicting an overall feature corresponding to all of the plurality of joints of the customer.

7. The method of claim 1 , wherein the predefined range is −80 to 80 degrees relative to the front direction of the customer.

8. The method of claim 1 , wherein the camera is a 3D ORBBEC camera, and the step of obtaining the first 3D pose of the customer is performed by an ORBBEC 3D pose model.

9. The method of claim 1 , wherein the step of generating the second SMPL pose of the user comprises:
estimating a 2D pose of the customer from the 2D component of the 3D image;
converting the 2D pose to a second 3D pose using a temporal dilated convolutional model; and
generating the second SMPL pose from the second 3D pose using the machine learning model.

10. The method of claim 9 , wherein the first 3D pose or the second 3D pose includes 3D coordinates of 12 joints of the customer, and the 12 joints comprise: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

11. The method of claim 1 , wherein the step of obtaining the second SMPL pose of the user is performed using an end-to-end pose and shape estimation model.

12. The method of claim 11 , wherein the end-to-end pose and shape estimation model is graph convolutional mesh regression (CMR).

13. A system comprising a computing device and a camera connected to the computing device, wherein the computing device comprises a processor and a storage device storing computer executable code, and the computer executable code, when executed at the processor, is configured to:
instruct the camera to capture a three dimensional (3D) image of a customer;
obtain a first 3D pose of the customer in the 3D image;
generate, by performing a machine learning model on the first 3D pose, a first skinned multi-person linear model (SMPL) pose, wherein the first SMPL pose comprises a plurality of rotation matrixes representing rotations of the customer's joints and a global rotation matrix representing whole body rotation of the customer;
calculate an angle of the whole body rotation of the customer based on the global rotation matrix;
when the angle of the whole body rotation is in a predefined range relative to a front direction of the customer: construct an SMPL model using the first SMPL pose; and
when the angle of the whole body rotation is out of the predefined range: generate a second SMPL pose using two dimensional (2D) component of the 3D image, and construct the SMPL model using the second SMPL pose.

14. The system of claim 13 , wherein the computer executable code is further configured to dress a garment model onto the SMPL model, and display the SMPL model dressed in the garment model on a display connected to the computing device.

15. The system of claim 13 , wherein the machine learning model comprises:
a non-residual block comprising a first fully-connected layer, a first batchnorm layer, and a first rectified linear unit (ReLU) layer;
a first residual block connected to the non-residual block, comprising a second fully-connected layer, a second batchnorm layer, and a second ReLU layer;
a second residual block connected to the first residual block, comprising a third fully-connected layer, a third batchnorm layer, and a third ReLU layer;
a plurality of joint self-attention blocks connected to the second residual block, each of the plurality of the joint self-attention blocks predicting a feature corresponding to one of a plurality of joints of the customer; and
a global self-attention block connected to the second residual block, predicting an overall feature corresponding to all of the plurality of joints of the customer.

16. The system of claim 13 , wherein the predefined range is −80 to 80 degrees relative to the front direction of the customer.

17. The system of claim 13 , wherein the computer executable code is configured to generate the second SMPL pose by:
estimating a 2D pose of the customer from 2D component of the 3D image; and
converting the 2D pose to a second 3D pose using a temporal dilated convolutional model; and
generating the second SMPL pose from the second 3D pose using the machine learning model.

18. The system of claim 13 , wherein the computer executable code is configured to generate the second SMPL pose using an end-to-end pose and shape estimation model.

19. A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to:
instruct a camera to capture a three dimensional (3D) image of a customer;
obtain a first 3D pose of the customer in the 3D image;
generate, by performing a machine learning model on the first 3D pose, a first skinned multi-person linear model (SMPL) pose, wherein the first SMPL pose comprises a plurality of rotation matrixes representing rotations of the customer's joints and a global rotation ma

Assignees

• Beijing Wodong Tianjun Information Technology Co Ltd
• Jd Com American Tech Corp

Inventors

• Fan Xiaochuan
• Miao Dan
• Lyu Chumeng

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title