Image style transfer for three-dimensional models

Patent metadata

Field	Value
Publication number	US-10467820-B2
Application number	US-201815878621-A
Country	US
Kind code	B2
Filing date	Jan 24, 2018
Priority date	Jan 24, 2018
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

Example aspects of the present disclosure are directed to systems and methods that perform image style transfer for three-dimensional models. In some implementations, the systems and methods can use machine-learned models such as, for example, convolutional neural networks to generate image style and content information used to perform style transfer. The systems and methods of the present disclosure can operate in a rendered image space. In particular, a computing system can iteratively modify an attribute rendering map (e.g., texture map, bump map, etc.) based on information collected from a different rendering of the model at each of a plurality of iterations, with the end result being that the attribute rendering map mimics the style of one or more reference images in content-preserving way. In some implementations, a computation of style loss at each iteration can be performed using multi-viewpoint averaged scene statistics, instead of treating each viewpoint independently.

First claim

What is claimed is:

1. A computer-implemented method, the method comprising:
obtaining, by one or more computing devices, one or more references images and data descriptive of a three-dimensional model;
determining, by the one or more computing devices, one or more reference image statistics for the one or more reference images; and
for each of a plurality of iterations:
rendering, by the one or more computing devices, a current image of the three-dimensional model from a current viewpoint using a current attribute rendering map, wherein the current attribute rendering map provides a plurality of attribute values respectively for a plurality of locations of the three-dimensional model;
determining, by the one or more computing devices, one or more current image statistics for the current image of the three-dimensional model rendered from the current viewpoint using the current attribute rendering map;
updating, by the one or more computing devices, one or more average image statistics based at least in part on the one or more current image statistics determined for the current image;
evaluating, by the one or more computing devices, a loss function that evaluates a difference between the one or more reference image statistics and the one or more average image statistics; and
modifying, by the one or more computing devices, the current attribute rendering map based at least in part on a gradient of the loss function.

2. The computer-implemented method of claim 1 , wherein the current attribute rendering map comprises a texture map.

3. The computer-implemented method of claim 1 , wherein the current attribute rendering map comprises one or more of:
a bump map;
a light color map;
a light position map; or
a material property map.

4. The computer-implemented method of claim 1 , further comprising, for each of the plurality of iterations and prior to said rendering, selecting, by the one or more computing devices, the current viewpoint for such iteration from a set of expected viewpoints.

5. The computer-implemented method of claim 1 , wherein the one or more average image statistics that are updated are an average of the image statistics determined in one or more previous iterations.

6. The computer-implemented method of claim 1 , comprising, after performance of a final iteration, storing or outputting the current attribute rendering map for use in rendering the three dimensional model.

7. The computer-implemented method of claim 1 , wherein:
determining, by the one or more computing devices, the one or more reference image statistics for the one or more reference images comprises:
inputting, by the one or more computing devices, the one or more reference images into a convolutional neural network; and
determining, by the one or more computing devices, the one or more reference image statistics based at least in part on one or more first Gram or covariance matrices computed over internal features of the convolutional neural network; and

for each of the plurality of iterations, determining, by the one or more computing devices, the one or more current image statistics for the current image of the three-dimensional model comprises:
inputting, by the one or more computing devices, the current image into the convolutional neural network; and
determining, by the one or more computing devices, the one or more current image statistics based at least in part on one or more second Gram or covariance matrices computed over internal features of the convolutional neural network.

8. The computer-implemented method of claim 1 , further comprising, for each of the plurality of iterations:
rendering, by the one or more computing devices, a base image of the three-dimensional model from the current viewpoint using a base attribute rendering map;
determining, by the one or more computing devices, one or more base content descriptors for the base image of the three-dimensional model rendered from the current viewpoint using the base attribute rendering map; and
determining, by the one or more computing devices, one or more current content descriptors for the current image of the three-dimensional model rendered from the current viewpoint using the current attribute rendering map;
wherein the loss function further evaluates a second difference between the one or more base content descriptors and the one or more current content descriptors.

9. The computer-implemented method of claim 8 , wherein:
determining, by the one or more computing devices, the one or more base content descriptors for the base image comprises:
inputting, by the one or more computing devices, the base image into a convolutional neural network; and
obtaining, by the one or more computing devices, a first embedding from a hidden layer of the convolutional neural network, the one or more base content descriptors comprising the first embedding; and

determining, by the one or more computing devices, the one or more current content descriptors for the current image comprises:
inputting, by the one or more computing devices, the current image into the convolutional neural network; and
obtaining, by the one or more computing devices, a second embedding from the hidden layer of the convolutional neural network, the one or more current content descriptors comprising the second embedding.

10. The computer-implemented method of claim 1 , wherein modifying, by the one or more computing devices, the current attribute rendering map based at least in part on the gradient of the loss function comprises backpropagating, by the one or more computing devices, the gradient of the loss function through a render operation performed by the one or more computing devices during said rendering of the current image.

11. The computer-implemented method of claim 1 , wherein modifying, by the one or more computing devices, the current attribute rendering map based at least in part on the gradient of the loss function comprises modifying, by the one or more computing devices, one or more pixel values of the current attribute rendering map based at least in part on the gradient of the loss function.

12. The computer-implemented method of claim 1 , wherein modifying, by the one or more computing devices, the current attribute rendering map based at least in part on the gradient of the loss function comprises modifying, by the one or more computing devices, a Laplacian pyramid of the current attribute rendering map based at least in part on the gradient of the loss function.

13. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations, the operations comprising:
determining one or more reference image statistics for one or more reference images; and
for each of a plurality of iterations:
obtaining a current image of a three-dimensional model rendered from a current viewpoint using a current attribute rendering map;
determining one or more current image statistics for the current image of the three-dimensional model rendered from the current viewpoint using the current attribute rendering map;
evaluating a loss function that provides a loss value based at least in part on the one or more reference image statistics and based at least in part on the one or more current image statistics; and
modifying, based at least in part on a gradient of the loss function, one or more of:
the current attribute rendering map;
one or more parameters of a map generation function configured to generate the current attribute render

Assignees

• Google Llc

Inventors

• Mordvintsev Alexander
• Sharifi Matthew

Classifications

• G06T15/20 Primary

  Perspective computation · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T15/00

  Three-dimensional [3D] image rendering · CPC title