Data driven method for transferring indoor scene layout and color style

Patent metadata

Field	Value
Publication number	US-10049297-B1
Application number	US-201715464209-A
Country	US
Kind code	B1
Filing date	Mar 20, 2017
Priority date	Mar 20, 2017
Publication date	Aug 14, 2018
Grant date	Aug 14, 2018

Abstract

The invention provides a data driven method for transferring indoor scene layout and color style, including: preprocessing images in an indoor image data set, which includes manually labeling semantic information and layout information; obtaining indoor layout and color rules on the data set by learning algorithms; performing object-level semantic segmentation on input indoor reference image, or performing object-level and component-level segmentations using color segmentation methods, to extract layout constraints and color constraints of reference images, associating the reference images with indoor 3D scene via the semantic information; constructing a graph model for indoor reference image scene and indoor 3D scene to express indoor scene layout and color; performing similarity measurement on the indoor scene and searching for similar images in the data set to obtain an image sequence with gradient layouts from reference images to input 3D scene; performing image-sequence-guided layout and color transfer generation for indoor 3D scene.

First claim

What is claimed is:

1. A data driven method for transferring an indoor scene layout and color style, comprising:
step 1, preprocessing an image in an indoor image data set, which comprises: manually labeling semantic information to obtain a semantic marking map for each image, and manually labeling layout information to obtain a top view of a layout map for each image;
step 2, obtaining an indoor layout rule and a color rule on the indoor image data set by a learning algorithm;
step 3, performing an object-level semantic segmentation on an input indoor reference image, or performing an object-level segmentation and a component-level segmentation using a color segmentation method, to extract a layout constraint and a color constraint of the reference image, and associating the reference image with an indoor three-dimensional (3D) scene via the semantic information;
step 4, constructing a graph model for an indoor reference image scene and the indoor 3D scene, and using the graph model to express a layout and color of an indoor scene;
step 5, performing, according to the graph model, similarity measurement on the indoor scene, and searching for a similar image in the data set to obtain an image sequence with a gradient layout from the reference image to the input 3D scene; and
step 6, performing, according to the layout rule and the color rule and combined with the layout constraint and the color constraint for each image in the gradient sequence, image-sequence-guided layout and color transfer generation for the indoor 3D scene to obtain an indoor 3D scene sequence with a layout and color style similar to that in the gradient image sequence.

2. The method according to claim 1 , wherein the color segmentation method in step 3 comprises that if different components of an object in an image have different color, the component-level segmentation is performed on the image by differentiating the different components using a color classification method.

3. The method according to claim 1 , wherein the graph model in step 4 is a tree structure graph which represents the indoor scene as a multi-layer structure containing nodes from three layers, wherein an entire scene is a global root node; an indoor area is divided into five regions which are region front, region back, region left, region right, region center, all serving as regional sub-nodes; an object contained in each region is an object leaf node of said region; wherein the graph model represents indoor scene layout distribution and an object pairing relation.

4. The method according to claim 1 , wherein performing similarity measurement on the indoor scene in step 5 comprises: comparing graph models of two indoor scenes, calculating similarity between different regions of the indoor scenes by calculating node similarity and side similarity of the graph models, and finally calculating overall similarity between the two indoor scenes.

5. The method according to claim 1 , wherein the image-sequence-guided layout and color transfer generation in step 6 is a method for gradient style transfer which is described specifically as: a style of image i in the image sequence being S i , a 3D scene after performing the style transfer according to image i being M i ; after completing the style transfer of an ith image, calculating a style difference between an (i+1)th image and the ith image, transferring the style difference to the 3D scene M i to generate a new 3D scene M i+1 , a formula being as follows:
M i+1 =M i ( S i+1 −S i );
wherein after such iteration, image-sequence-guided indoor 3D scene transfer generation is completed.

Assignees

• Univ Beihang

Inventors

• Chen Xiaowu
• Li Jianwei
• Li Qing
• Zou Dongqing
• Gao Bo
• Zhao Qinping

Classifications

• G06T17/005 Primary

  Tree description, e.g. octree, quadtree · CPC title

• G06T2210/61

  Scene description · CPC title

• G06K9/6206 Primary

  Physics · mapped topic

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T7/11

  Region-based segmentation · CPC title