Method of fitting virtual item using human body model and system for providing fitting service of virtual item

Patent metadata

Field	Value
Publication number	US-9607419-B2
Application number	US-201314106532-A
Country	US
Kind code	B2
Filing date	Dec 13, 2013
Priority date	Dec 14, 2012
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

A method of fitting a virtual item using a human body model and a system for providing a fitting service of a virtual item are provided, the method including determining whether a user is located in an experience area, loading a three-dimensional (3D) standard avatar corresponding to setting information input by the user when the user is located in the trial area, obtaining a depth image of the user corresponding to a preset posture, transforming the 3D standard avatar into a user avatar reflecting body characteristics of the user, using a depth image of the user, fitting a virtual item selected by the user to the user avatar, and applying a motion of the user changing in real time to the user avatar to which the virtual item is fitted.

First claim

What is claimed is:

1. A method of fitting a virtual item, the method comprising:
determining whether a user is located in an experience area;
loading a three-dimensional (3D) standard avatar corresponding to setting information input by the user when the user is located in the experience area;
obtaining a depth image of the user corresponding to a preset posture;
transforming the 3D standard avatar into a user avatar reflecting body characteristics of the user using the depth image of the user;
fitting a virtual item selected by the user to the user avatar;
applying a motion of the user changing in real time to the user avatar to which the virtual item is fitted;
calculating transformation of the virtual item when the motion of the user is applied to the user avatar;
synthesizing a result of rendering of the transformed virtual item with a real color image of the user captured by a camera; and
displaying a result image rendered based on the synthesized result,
wherein the calculating comprises reverse-reflecting calculated difference of top and bottom of shoulder in a result of retargeting with respect to motion of the user avatar, and controlling motion information of a joint to offset difference error.

2. The method of claim 1 , wherein the transforming of the 3D standard avatar into the user avatar reflecting the appearance of the body characteristics of the user comprises:
restoring a 3D shape patch from the depth image of the user;
changing joint length information of the 3D standard avatar, using the 3D shape patch; and
changing volume information of the 3D standard avatar using the 3D shape patch.

3. The method of claim 2 , wherein the restoring of the 3D shape patch comprises:
representing the depth image in a 3D mesh using depth information of a current pixel and an adjacent pixel of the depth image; and
determining a 3D position of a vertex configuring the 3D mesh based on the depth information and calibration information.

4. The method of claim 2 , wherein the changing of the joint length information of the 3D standard avatar comprises:
estimating a joint structure of the user using the 3D shape patch; and
extracting joint length information of the user from the estimated joint structure, and applying the extracted joint length information to the joint length information of the 3D standard avatar.

5. The method of claim 2 , wherein the changing of the volume information of the 3D standard avatar comprises:
generating a cross section vertical to a progressive direction of a joint from the joint structure of the user estimated from the 3D shape patch; and
adjusting the volume information of the 3D standard avatar using an intersection of the restored 3D shape patch and the cross section.

6. The method of claim 1 , wherein the fitting of the virtual item selected by the user to the user avatar comprises:
determining a displacement between a mesh of the 3D standard avatar and a mesh of the user avatar; and
applying the determined displacement to a vertex of the virtual item.

7. The method of claim 1 , wherein the applying of the detected motion of the user changing in real time to the user avatar fitted with the virtual item comprises:
retargeting a motion of the user avatar based on 3 degrees of freedom (DOF) rotation and 3 DOF movement of a joint configuring the joint structure of the user avatar.

8. The method of claim 1 , wherein the applying of the detected motion of the user changing in real time to the user avatar fitted with the virtual item comprises:
rendering lighting effects to the virtual item based on the motion of the user avatar.

9. The method of claim 1 , further comprising:
synthesizing the user avatar fitted with the virtual item with the color image of the user, and providing the user avatar fitted with the virtual item.

10. A method of fitting a virtual item, the method comprising:
providing an interface for selecting a virtual item to a user located in an experience area for a fitting service of a virtual item;
displaying a user avatar fitted with a virtual item selected by the user;
calculating transformation of the virtual item when the motion of the user is applied to the user avatar;
synthesizing a result of rendering of the transformed virtual item with a real color image of the user captured by a camera;
displaying a result image rendered based on the synthesized result;
generating an identification tag associated with purchase information or figure information on a virtual item when the user selects the displayed user avatar; and
providing a captured screen of the user avatar, overlaid with the generated identification tag,
wherein the calculating comprises reverse-reflecting calculated difference of top and bottom of shoulder in a result of retargeting with respect to motion of the user avatar, and controlling motion information of a joint to offset difference error.

11. The method of claim 10 , wherein the displaying of the user avatar fitted with the virtual item selected by the user comprises:
transforming a three-dimensional (3D) standard avatar corresponding to setting information input by the user into the user avatar reflecting body characteristics of the user, and displaying the transformed user avatar.

12. The method of claim 11 , wherein the user avatar corresponds to an avatar of which joint length information and volume information of the 3D standard avatar are changed based on the body characteristics of the user.

13. A system for providing a fitting service of a virtual item, the system comprising:
a display to display an experience image of a user to provide a fitting service of a virtual item to the user;
an apparatus for capturing a color image to capture the user in real time, and collect a color image of the user;
an apparatus for capturing a depth image to measure a motion and a body size of the user;
an apparatus for fitting a virtual item to convert a three-dimensional (3D) standard avatar into a user avatar based on the motion and the body size of the user, and fit a virtual item to the user avatar;
an apparatus for calculating transformation of the virtual item when the motion of the user is applied to the user avatar;
an apparatus for synthesizing a result of rendering of the transformed virtual item with a real color image of the user captured by a camera; and
an apparatus for displaying a result image rendered based on the synthesized result,
wherein the apparatus for fitting the virtual item synthesizes the user avatar fitted with the virtual item with the color image, and transmits the user avatar fitted with the virtual item to the display, and
the apparatus for calculating transformation comprises reverse-reflects calculated difference of top and bottom of shoulder in a result of retargeting with respect to motion of the user avatar, and controlling information of a joint to offset difference error.

14. The system of claim 13 , wherein the apparatus for fitting the virtual item changes joint length information and volume information of the 3D standard avatar using a 3D shape patch restored from a depth image of the user.

15. The system of claim 13 , wherein the apparatus for fitting the virtual item retargets a motion of the user avatar based on 3 degrees of freedom (DOF) rotation and 3 DOF movement of a joint configuring a joint structure of the user avatar.

16. The system of claim 13 , wherein the apparatus for fitting the virtual item renders lighting effects to the virtual item based on the motion of the user avatar.

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Kim Ho Won
• Lee Seung Wook
• Kim Ki Nam
• Choi Jin Sung
• Koo Bon Ki
• Cho Kyu Sung
• Kim Tae Joon
• Sohn Sung Ryull
• Kim Do Hyung

Classifications

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06T2210/16

  Cloth · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title