Transformation of hand-drawn sketches to digital images

Patent metadata

Field	Value
Publication number	US-11048932-B2
Application number	US-201916551025-A
Country	US
Kind code	B2
Filing date	Aug 26, 2019
Priority date	Aug 26, 2019
Publication date	Jun 29, 2021
Grant date	Jun 29, 2021

Abstract

Techniques are disclosed for generating a vector image from a raster image, where the raster image is, for instance, a photographed or scanned version of a hand-drawn sketch. While drawing a sketch, an artist may perform multiple strokes to draw a line, and the resultant raster image may have adjacent or partially overlapping salient and non-salient lines, where the salient lines are representative of the artist's intent, and the non-salient (or auxiliary) lines are formed due to the redundant strokes or otherwise as artefacts of the creation process. The raster image may also include other auxiliary features, such as blemishes, non-white background (e.g., reflecting the canvas on which the hand-sketch was made), and/or uneven lighting. In an example, the vector image is generated to include the salient lines, but not the non-salient lines or other auxiliary features. Thus, the generated vector image is a cleaner version of the raster image.

First claim

What is claimed is:

1. A method for generating an output vector image from a raster image of a sketch, the method comprising:
receiving an input raster image of the sketch, the input raster image comprising a first line that is partially overlapping and adjacent to a second line, and one or both of a non-white background on which the plurality of lines are drawn and a non-uniform lighting condition;
identifying, by a deep learning network and in the input raster image, the first line as a salient line, and the second line as a non-salient line;
generating, by the deep learning network, an intermediate raster image that includes the first line, but not the second line, and one or both of a white background and a uniform lighting condition; and
converting the intermediate raster image to the output vector image.

2. The method of claim 1 , wherein the input raster image further includes one or more of a blemish, a defect, a watermark, and/or the non-white background, and the intermediate raster image does not include any of the blemish, defect, watermark, or non-white background.

3. The method of claim 1 , wherein the input raster image includes both of the non-white background and the non-uniform lighting condition, and the intermediate raster image includes both of the white background and the uniform lighting condition.

4. The method of claim 1 , wherein prior to receiving the input raster image, the method further comprises training the deep learning network, the training comprising:
synthesizing a plurality of training raster images from a plurality of training vector images; and
training the deep learning network using the plurality of training raster images and the plurality of training vector images.

5. The method of claim 1 , wherein prior to receiving the input raster image, the method further comprises training the deep learning network, the training comprising:
generating training data for training the deep learning network, wherein generating the training data comprises
generating a sketch style dataset comprising a plurality of image pairs, each image pair including a vector image and a corresponding raster image, wherein the raster image of each image pair is a scanned or photographed version of a corresponding hand-drawn sketch that mimics the vector image of that image pair, and
synthesizing a training raster image from a corresponding training vector image, based at least in part on a stroke style analogy of an image pair of the sketch style dataset, such that the synthesized training raster image has a stroke style that that mimics a stroke style of the image pair of the sketch style dataset; and

training the deep learning network using the training data.

6. The method of claim 1 , wherein the deep learning network comprises a generator and a discriminator, wherein the generator uses a residual block architecture, and the discriminator uses a generative adversarial network (GAN).

7. The method of claim 6 , wherein the generator comprises:
a down-sampler comprising a first plurality of convolution layers;
a transformer comprising a second plurality of convolution layers; and
an up-sampler comprising a third plurality of convolution layers.

8. The method of claim 6 , wherein the generator is trained using training data comprising (i) a plurality of training vector images and (ii) a plurality of training raster images synthesized from the plurality of training vector images, and wherein prior to receiving the input raster image, the method further comprises:
generating a loss function to train the generator, the loss function comprising one or more of
a pixel loss that is based at least in part on per-pixel L 1 norm of difference between (i) a first training vector image of the plurality of training vector images, and (ii) a second vector image generated by the deep learning network from a first training raster image of the plurality of training raster images, wherein the first training raster image of the training data is synthesized from the first training vector image of the training data, the first training vector image being a ground truth image,
an adversarial loss based on a hinge loss of the discriminator, and/or
a minimum (Min)-pooling loss.

9. The method of claim 8 , wherein generating the loss function comprises generating the Min-pooling loss by:
assigning, to individual pixels in each of the first training vector image and the second vector image, a value of “1” for background and a value of “0” for inked curve; and
subsequent to assigning the values, applying min-pooling one or more times to each of the first training vector image and the second vector image.

10. The method of claim 9 , wherein generating the loss function comprises generating the Min-pooling loss by:
subsequent to applying the min-pooling, generating the Min-pooling loss based on a L 1 distance between the first vector image and the second vector image.

11. A system for converting an input raster image to a vector image, the input raster image including a plurality of salient lines and a plurality of auxiliary features, the system comprising:
one or more processors;
a raster-to-raster conversion module executable by the one or more processors to receive the input raster image, and generate an intermediate raster image that includes the plurality of salient lines and lacks the plurality of auxiliary features; and
a raster-to-vector conversion module executable by the one or more processors to convert the intermediate raster image to the output vector image.

12. The system of claim 11 , wherein the raster-to-raster conversion module comprises the deep learning network that includes a generator and a discriminator, wherein the generator uses a residual block architecture, and the discriminator uses a generative adversarial network (GAN).

13. The system of claim 12 , wherein the generator comprises:
a down-sampler comprising a first plurality of convolution layers and to recognize salient features, auxiliary features, and/or a non-white background of the input raster image at one or more image resolutions;
a transformer having an encoder-decoder framework comprising a second plurality of convolution layers and to perform processing of features recognized by the down-sampler at a higher dimension space; and
an up-sampler comprising a third plurality of convolution layers and to restore resolution to that of the input raster image.

Assignees

• Adobe Inc

Inventors

• Batra Vineet
• Fisher Matthew David
• Phogat Ankit

Classifications

• G06T9/00 Primary

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V30/333

  Preprocessing; Feature extraction · CPC title

• G06T11/23

  using straight lines or curves · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title