Combined Structure and Style Network

Patent metadata

Field	Value
Publication number	US-2018357519-A1
Application number	US-201715616776-A
Country	US
Kind code	A1
Filing date	Jun 7, 2017
Priority date	Jun 7, 2017
Publication date	Dec 13, 2018
Grant date	—

Abstract

A combined structure and style network is described. Initially, a large set of training images, having a variety of different styles, is obtained. Each of these training images is associated with one of multiple different predetermined style categories indicating the image's style and one of multiple different predetermined semantic categories indicating objects depicted in the image. Groups of these images are formed, such that each group includes an anchor image having one of the styles, a positive-style example image having the same style as the anchor image, and a negative-style example image having a different style. Based on those groups, an image style network is generated to identify images having desired styling by recognizing visual characteristics of the different styles. The image style network is further combined, according to a unifying training technique, with an image structure network configured to recognize desired objects in images irrespective of image style.

First claim

What is claimed is:

1 . In a digital medium environment to identify images having a desired object and a desired styling, a method implemented by at least one computing device, the method comprising:
generating, by the at least one computing device, a combined structure and style network by combining an image structure network with an image style network, wherein:
the image structure network is configured to identify an object in an image based on one or more visual characteristics of a sketched image of the object;
the image style network is configured to identify a styling in the image based on another one or more visual characteristics of an image having the styling; and
the image structure and style networks are combined using a unifying training technique that exposes the image structure and style networks to groups of training images for learning both object and styling visual characteristics, at least one of the groups including one or more anchor training images having an example of the object and the styling, a positive training image having a different example of the object and the styling, and a negative training image having at least one of a different object or different styling from the one or more anchor training images; and
identifying, by the at least one computing device, both the object and the styling in the image by using the combined structure and style network for an image search; and outputting, by the at least one computing device, data indicative of the image with the identified object and styling.

2 . A method as described in claim 1 , further comprising generating the image style network by training the image style network to identify a plurality of different styles in images based on the styling visual characteristics of the training images having the plurality of different styles.

3 . A method as described in claim 2 , wherein training the image style network to identify the plurality of different styles includes, for a given style:
learning similar visual characteristics of the training images having the given style; and learning different visual characteristics between the training images having the given style and the training images having the other different styles.

4 . A method as described in claim 1 , wherein the image structure network and the image style network are both configured as triplet convolutional neural networks.

5 . A method as described in claim 1 , wherein the combined structure and style network is configured as a hierarchical triplet convolutional neural network comprising a structure stream based on the image structure network and a style stream based on the image style network.

6 . A method as described in claim 1 , further comprising generating the image structure network, in part, by training the image structure network to identify a plurality of different objects based on the object visual characteristics of example sketches and example photographic images having the plurality of different objects.

7 . A method as described in claim 1 , further comprising:
receiving a style-supplemented image request that includes additional data indicative of the sketched image of the object and the image having the styling; and identifying the object and the styling in the image responsive to the style-supplemented image request.

8 . A method as described in claim 1 , further comprising communicating the data indicative of the image to a client device that requested the image search.

9 . A method as described in claim 1 , wherein:
the image structure network is further configured to identify the object in the image based on an additional one or more visual characteristics of at least one different type of image of the object; and the unifying training technique further enabling the combined structure and style network to identify both the object and the styling in the image based on a style-supplemented image request that includes additional data indicative of the at least one different type of image of the object and the image having the styling.

10 . A method as described in claim 9 , wherein the at least one different type of image comprises at least one of:
a photographic image; or an artistically styled image.

11 . A system comprising:
at least one processor; and memory having stored thereon computer-readable instructions that are executable by the at least one processor to perform operations for identifying items of content having a desired structure of the content and a desired styling of the content, the operations comprising:
generating a combined structure and style network by combining a content structure network with a content style network, wherein:
the content structure network is configured to identify the structure in the content based on one or more content characteristics indicated by a description of the structure;
the content style network is configured to identify a styling in the content based on another one or more content characteristics of a content item having the styling; and
the content structure and style networks are combined using a unifying training technique that enables the combined structure and style network to identify both the structure and the styling in the content in connection with a content search; and

outputting data indicative of the content with the structure and the styling responsive to identification by the combined structure and style network in connection with the content search.

12 . A system as described in claim 11 , wherein the content is image content, the content item is an image, and the description of the structure comprises another image

13 . A system as described in claim 11 , wherein the content comprises a type of content different from image content.

14 . A system as described in claim 11 , wherein the description of the structure comprises a same type of content as the content item having the styling.

15 . A system as described in claim 11 , wherein the description of the structure comprises a different type of content from the content item having the styling.

16 . A system as described in claim 11 , wherein the description of the structure comprises text and the content item having the styling comprises an image.

17 . In a digital medium environment to identify images having a desired styling, a method implemented by at least one computing device, the method comprising:
obtaining, by the at least one computing device, a plurality of training images, each of the training images being associated with a style category of a plurality of predetermined style categories and a semantic category of a plurality of predetermined semantic categories; forming, by the at least one computing device, groups of the training images based on the associated style and semantic categories; generating, by the at least one computing device, digital content comprising an image style network configured to identify the images having a desired styling by training the image style network with the formed groups; and outputting, by the at least one computing device, data indicative of an image identified via the image style network as having the desired styling.

18 . A method as described in claim 17 , wherein each of the formed groups comprises:
an anchor image that is associated with a given style category of the plurality of predetermined style categories and a given semantic category of the plurality of predetermined semantic categories; a positive-style example image that is associated with the given style

Assignees

• Adobe Systems Inc

Inventors

• Jin Hailin
• Collomosse John Philip

Classifications

• G06V10/803

  of input or preprocessed data · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24323

  Tree-organised classifiers · CPC title

• G06F18/251

  of input or preprocessed data · CPC title