Image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-10366533-B2
Application number	US-201715420545-A
Country	US
Kind code	B2
Filing date	Jan 31, 2017
Priority date	Mar 20, 2015
Publication date	Jul 30, 2019
Grant date	Jul 30, 2019

Abstract

According to one embodiment, an image processing device includes at least one processor. The at least one processor is configured to acquire a first three-dimensional model regarding a subject, set a plurality of first control points on the first three-dimensional model, acquire mesh data of a meshed image of a region of clothing extracted from a captured image, acquire a second three-dimensional model, modify the mesh data based on an amount of movement from each of the plurality of first control points, to each respective one of a plurality of second control points, and generate an image of the clothing using the captured image and the modified mesh data.

First claim

What is claimed is:

1. An image processing device comprising at least one processor configured to:
acquire a first three-dimensional model regarding a subject;
set a plurality of first control points on the first three-dimensional model;
acquire mesh data of a mesh image of a region of clothing extracted from a captured image including the subject wearing the clothing, the acquired mesh data comprising a plurality of vertexes;
acquire a plurality of second three-dimensional models by modifying a direction, a posture, and/or a body type of the first three-dimensional model;
generate modified mesh data for each respective one of the plurality of second three-dimensional models based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points, respectively corresponding to the plurality of first control points, on the respective one of the plurality of second three-dimensional models; and
generate a plurality of images of the clothing using the captured image and the modified mesh data, wherein
the at least one processor is configured to:
calculate, as an influence degree for each of a plurality of respective first control point-vertex combinations, a distance between the first control point and the vertex of the respective combination in a total region of a region of the first three-dimensional model and a region of the clothing represented by the mesh data in a case in which the modified mesh data is superimposed on the first three-dimensional model to represent a state in which the subject wears the clothing; and
acquire boundary information indicating a boundary of the region of the clothing, the boundary being designated on the region of the clothing, and each distance is based on the shortest path which does not cross the boundary indicated by the boundary information.

2. The image processing device of claim 1 , wherein
the mesh data is modified using the calculated influence degrees.

3. The image processing device of claim 1 , wherein
the plurality of first control points set on the first three-dimensional model comprises a physical feature point in the first three-dimensional model.

4. The image processing device of claim 1 , wherein
the plurality of first control points is positioned on an outer periphery of the first three-dimensional model in a case in which the first three-dimensional model is projected.

5. The image processing device of claim 1 , wherein
the at least one processor is configured to acquire the first three-dimensional model regarding the subject captured by an image capture unit including a camera,
each of the plurality of first control points set on the first three-dimensional model contains depth information indicating a position of the first control point in a depth direction with respect to the image capture unit, and
the at least one processor is configured to:
calculate depth information corresponding to each of the plurality of vertexes of the acquired mesh data based on the depth information contained in each of the plurality of first control points set on the first three-dimensional model; and
perform a hidden surface elimination process based on depth information corresponding to each of the plurality of vertexes of the modified mesh data when the plurality of images of the clothing is generated.

6. An image processing method to be executed by an image processing device, comprising:
acquiring a first three-dimensional model regarding a subject;
setting a plurality of first control points on the acquired first three-dimensional model;
acquiring mesh data of a mesh image of a region of clothing extracted from a captured image including the subject wearing the clothing, the acquired mesh data comprising a plurality of vertexes;
acquiring a plurality of second three-dimensional models by modifying a direction, a posture, and/or a body type of the first three-dimensional model;
generating modified mesh data for each respective one of the plurality of second three-dimensional models based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points, respectively corresponding to the plurality of first control points, on the respective one of the plurality of second three-dimensional models; and
generating a plurality of images of the clothing using the captured image and the modified mesh data, wherein
the method further comprises:
calculating, as an influence degree for each of a plurality of respective first control point-vertex combinations, a distance between the first control point and the vertex of the respective combination in a total region of a region of the first three-dimensional model and a region of the clothing represented by the mesh data in a case in which the modified mesh data is superimposed on the first three-dimensional model to represent a state in which the subject wears the clothing; and
acquiring boundary information indicating a boundary of the region of the clothing, the boundary being designated on the region of the clothing, and

each distance is based on the shortest path which does not cross the boundary indicated by the boundary information.

7. The image processing method of claim 6 , wherein
the mesh data is modified using the calculated influence degrees.

8. The image processing method of claim 6 , wherein
the plurality of first control points set on the first three-dimensional model comprises a physical feature point in the first three-dimensional model.

9. The image processing method of claim 6 , wherein
the plurality of first control points are positioned on an outer periphery of the first three-dimensional model in a case in which the first three-dimensional model is projected.

10. The image processing method of claim 6 , further comprising:
calculating depth information,
wherein
the acquiring of the first three-dimensional model comprises acquiring the first three-dimensional model regarding the subject captured by an image capture unit including a camera,
each of the plurality of first control points set on the first three-dimensional model comprises depth information indicating a position of the first control point in a depth direction with respect to the image capture unit, and
the calculating of the depth information comprises calculating depth information corresponding to each of the plurality of vertexes of the acquired mesh data based on the depth information contained in each of the plurality of first control points set on the first three-dimensional model, and
the generating comprises performing a hidden surface elimination process based on depth information corresponding to each of the plurality of vertexes of the modified mesh data when the plurality of images of the clothing is generated.

Assignees

• Toshiba Kk

Inventors

• Sugita Kaoru
• Nishiyama Masashi
• Sekine Masahiro
• Ohira Hidetaka

Classifications

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T2210/16

  Cloth · CPC title