Method and system for clothing virtual try-on service based on deep learning

Patent metadata

Field	Value
Publication number	US-12165196-B2
Application number	US-202217703903-A
Country	US
Kind code	B2
Filing date	Mar 24, 2022
Priority date	Mar 26, 2021
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

A method of a clothing virtual try-on service based on deep learning allowing a virtual try-on service application executed by at least one processor of a computing device to perform a process of the clothing virtual try-on service based on the deep-learning includes: determining a first clothes image including a first clothes object, a second clothes image including a second clothes object, and a model image including a model object; generating a first transformed clothes image by transforming a shape of the first clothes object to correspond to the model object; generating a second transformed clothes image by transforming a shape of the second clothes object to correspond to the model object; and generating and outputting a virtual fitting image obtained by synthesizing the first transformed clothes image and the second transformed clothes image to be virtually fitted on the model object.

First claim

What is claimed is:

1. A method for a clothing virtual try-on service based on deep learning and performed by a virtual try-on service application executed by at least one processor of a computing device, the method comprising:
determining a first clothes image including a first clothes object, a second clothes image including a second clothes object, and a model image including a model object;
generating a first transformed clothes image by transforming a shape of the first clothes object included in the first clothes image to correspond to the model object included in the model image;
generating a second transformed clothes image by transforming a shape of the second clothes object included in the second clothes image to correspond to the model object included in the model image; and
generating and outputting a virtual fitting image obtained by synthesizing the first transformed clothes image and the second transformed clothes image to be virtually fitted on the model object,
wherein the method further comprises segmenting the model object into a plurality of areas based on the model image, and generating a clothing guide map (CGMap) in which label information is determined for each of the segmented areas of the model object,
wherein the segmenting of the model object into the plurality of areas based on the model image includes:
segmenting a first fitting area for the first clothes object based on the model image and the first clothes object of the first clothes image;
segmenting a second fitting area for the second clothes object based on the second clothes object of the second clothes image; and
determining an overlapping area where a predetermined body area and the first and/or second fitting area of the model object are overlapped with each other and determining the determined overlapping area to be one of the predetermined body area and the first and/or second fitting area.

2. The method of claim 1 , wherein the generating of the first transformed clothes image includes determining a segmentation area of the clothing guide map (CGMap) matching the first clothes object, and transforming the first clothes object by twisting and aligning the first clothes object to be geometrically matched with the determined segmentation area.

3. The method of claim 1 , wherein the generating of the first transformed clothes image includes:
determining a first geometric matching module that, when the first clothes object is a top garment, transforms the top garment; and
generating a 1-1th transformed clothes object image by twisting and aligning, by the determined first geometric matching module, a direction of the first clothes object to be matched with a direction of an upper body area of the model object.

4. The method of claim 3 , wherein the generating of the first transformed clothes image includes:
determining, by the determined first geometric matching module, an overlapping area overlapped with the upper body area among body areas of the model object and an adjacent area adjacent to the upper body area; and
generating the first transformed clothes image by transforming the 1-1th transformed clothes object image according to the overlapping area overlapped with the upper body area and the adjacent area adjacent to the upper body area.

5. The method of claim 4 , wherein the generating of the second transformed clothes image includes:
determining a second geometric matching module that, when the second clothes object is a bottom garment, transforms the bottom garment;
generating a 2-1th transformed clothes object image by twisting and aligning, by the determined second geometric matching module, a direction of the second clothes object to be matched with a direction of a lower body area of the model object;
determining, by the determined second geometric matching module, an overlapping area overlapped with the lower body area among the body areas of the model object and an adjacent area adjacent to the lower body area; and
generating the second transformed clothes image by transforming the 2-1th transformed clothes object image according to the overlapping area overlapped with the lower body area and the adjacent area adjacent to the lower body area.

6. The method of claim 5 , wherein the generating and outputting of the virtual fitting image obtained by synthesizing the first transformed clothes image and the second transformed clothes image to be virtually fitted on the model object includes:
generating a transformed clothes object in which the first transformed clothes image and the second transformed clothes image are aligned to be matched with the model object; and
generating the virtual fitting image in which the transformed clothes object is virtually fitted on the model object.

7. The method of claim 1 , wherein the generating of the first transformed clothes image includes transforming an outline shape of the first clothes object to correspond to a outline shape of a first body area of the model object.

8. The method of claim 1 , wherein the generating of the first transformed clothes image further includes correcting an inner shape of the transformed first clothes object according to an inner shape of the first clothes object before the inner shape of the transformed first clothes object is transformed.

9. The method of claim 1 , wherein the generating and outputting of the virtual fitting image obtained by synthesizing the first transformed clothes image and the second transformed clothes image to be virtually fitted on the model object further includes generating an intermediate model image based on the first and/or second transformed clothes object and the model image.

10. The method of claim 9 , wherein the generating of the intermediate model image based on the first and/or second transformed clothes object and the model image includes generating the intermediate model image by virtually generating a part of a body of the model object according to the first and/or second transformed clothes object corresponding to the model object of the model image.

11. The method of claim 10 , wherein the generating of the intermediate model image by virtually generating the part of the body of the model object according to the first and/or second transformed clothes object corresponding to the model object of the model image includes virtually generating a first body corresponding to the first transformed clothes object of the first clothes image, and virtually generating a second body corresponding to the second transformed clothes object of the second clothes image.

12. The method of claim 11 , wherein the generating and outputting of the virtual fitting image obtained by synthesizing the first transformed clothes image and the second transformed clothes image to be virtually fitted on the model object includes generating the virtual fitting image based on the intermediate model image, the first and/or second transformed clothes object, and a clothing guide map (CGMap).

13. A system for a clothing virtual try-on service based on deep learning, comprising:
at least one processor; and
at least one memory,
wherein at least one application stored in the memory and executed by the at least one processor to perform a process of the clothing virtual try-on service based on the deep learning is configured to:
determine a first clothes image including a first clothes object, a second clothes image including a second clothes object, and a model image including a model object,
generate the first transformed clothes image by transforming a shape of the first clothes object included in the first clothes image to correspond to the model object included

Assignees

• Nhn Corp

Inventors

• Lee Rokkyu
• Kang Minseok
• Koo Byumhyuk
• Lee Hyugjae

Classifications

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T7/12

  Edge-based segmentation · CPC title

• G06T7/38

  Registration of image sequences · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title