Methods and devices for vector line drawing

Patent metadata

Field	Value
Publication number	US-11928759-B2
Application number	US-202217724452-A
Country	US
Kind code	B2
Filing date	Apr 19, 2022
Priority date	Apr 19, 2022
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

The present disclosure describes methods and devices for generating a vector line drawing. A vector line drawing network may include a machine learning-based model that is trained to convert a raster image to a vector line drawing directly. The vector line drawing network may be trained end-to-end, using supervised learning, where only raster images are used as training data. A vector line drawing is generated stroke by stroke, over a series of time steps. In each time step, a dynamic drawing window is moved and scaled across the input raster image to sample a patch of the raster image, and a drawing stroke is predicted to draw a stroke in a corresponding patch in the canvas for the vector line drawing. The image patches are pasted in the canvas to assemble a final vector line drawing that corresponds to the input raster image.

First claim

The invention claimed is:

1. A method of generating vector line drawings comprising:
obtaining an input image, the input image being a raster image of a ground-truth drawing;
initiating a blank canvas image equal in size to the input image;
using a vector line drawing network, generating a vector line drawing over a series of time steps by, for each time step in the series of time steps:
obtaining a defined window position and a defined window size and defining a dynamic drawing window accordingly;
obtaining an image patch from the input image;
obtaining a canvas patch from the canvas image;
the image patch and the canvas patch being obtained from the input image and the canvas image, respectively, based on the defined window position and defined window size of the dynamic drawing window, by:
resampling the dynamic drawing window into spatial bins based on the defined window size of the dynamic drawing window and a resampling size;
setting sampling points inside each bin;
computing values for the sampling points by bilinear image interpolation; and
extracting the image patch and the canvas patch from the input image and the canvas image, respectively;

generating, using a stroke generator, a stroke action for a current time step, wherein the image patch and the canvas patch are provided as inputs to the stroke generator, the stroke action including positional values for computing a set of stroke parameters;
defining the defined window position and the defined window size for a next time step based on the stroke action generated for the current time step;
rendering a rendered stroke image using the set of stroke parameters computed from the positional values; and
pasting the rendered stroke image on to the canvas image based on the position and size of the dynamic drawing window; and
outputting the generated vector line drawing.

2. The method of claim 1 , wherein the stroke action for the current time step is a vector containing one or more action parameters for a predicted drawing stroke for the current time step in the series of time steps, the action parameters including at least one of:
an offset with respect to a current position of a virtual pen;
an intermediate control point;
a width factor, the width factor describing the line thickness for the predicted drawing stroke;
a scaling factor, the scaling factor defining the defined size of the dynamic window; or
a pen state parameter of the virtual pen.

3. The method of claim 2 , wherein the pen state parameter indicates whether the predicted drawing stroke is drawn in the canvas or whether the predicted drawing stroke is a movement of the dynamic drawing window to a different region of the input image without drawing the predicted drawing stroke in the canvas.

4. The method of claim 3 , wherein the predicted drawing stroke is a curve, the curve defined by one or more stroke parameters computed from the stroke action.

5. The method of claim 1 , wherein pasting the rendered stroke image on to the canvas image based on the position and size of the dynamic drawing window comprises:
defining a secondary coordinate system for the rendered stroke image;
resampling the rendered stroke image into spatial bins based on the size of the dynamic drawing window and a resampling size;
setting sampling points inside each bin;
computing values for the sampling points by bilinear image interpolation; and
pasting the resampled rendered stroke image on to the canvas image.

6. The method of claim 5 , wherein the vector line drawing network is a recurrent neural network, the recurrent neural network producing a plurality of consecutive drawing strokes, the consecutive drawing strokes being ordered by the order in which a respective rendered stroke image is pasted on to the canvas image.

7. The method of claim 1 , wherein end-to-end training of the vector line drawing network includes the calculation of an overall loss function, the overall loss function comprising at least one of:
the raster loss for visual raster-only supervision;
an out-of-bounds penalty loss; or
a stroke regularization loss.

8. The method of claim 7 , wherein calculating the raster loss includes the calculation of a normalized perceptual loss function, the perceptual loss function comparing a rendered line drawing and the target line drawing for a set of layers associated with a differentiable renderer.

9. The method of claim 7 , wherein calculating the stroke regularization loss includes calculating a stroke regularization term, the stroke regularization term being proportional to the number of drawn strokes, the stroke regularization loss acting to minimize the total number of drawn strokes by encouraging the production of longer drawing strokes over shorter drawing strokes and discouraging the production of redundant drawing strokes.

10. A device for generating vector line drawings, the device comprising a processor configured to execute instructions to cause the device to:
obtain an input image, the input image being a raster image of a ground-truth drawing;
initiate a blank canvas image equal in size to the input image;
using a vector line drawing network, generate a vector line drawing over a series of time steps by, for each time step in the series of time steps:
obtain a defined window position and a defined window size and defining a dynamic drawing window accordingly;
obtain an image patch from the input image;
obtain a canvas patch from the canvas image;
the image patch and the canvas patch being obtained from the input image and the canvas image, respectively, based on the defined window position and defined window size of the dynamic drawing window, by:
resampling the dynamic drawing window into spatial bins based on the defined window size of the dynamic drawing window and a resampling size;
setting sampling points inside each bin;
computing values for the sampling points by bilinear image interpolation; and
extracting the image patch and the canvas patch from the input image and the canvas image, respectively;

generate, using a stroke generator, a stroke action for a current time step, wherein the image patch and the canvas patch are provided as inputs to the stroke generator, the stroke action including positional values for computing a set of stroke parameters;
define the defined window position and the defined window size for a next time step based on the stroke action generated for the current time step;
render a rendered stroke image using the set of stroke parameters computed from the positional values; and
paste the rendered stroke image on to the canvas image based on the position and size of the dynamic drawing window; and

output the generated vector line drawing.

11. The device of claim 10 , wherein the stroke action for the current time step is a vector containing one or more action parameters for a predicted drawing stroke for the current time step in the series of time steps, the action parameters including at least one of:
an offset with respect to a current position of a virtual pen;
an intermediate control point;
a width factor, the width factor describing the line thickness for the predicted drawing stroke;
a scaling factor, the scaling factor defining the defined size of the dynamic window; or
a pen state parameter of the virtual pen.

12. The device of claim 11 , wherein the pen state parameter indicates whether the predicted drawing stroke is drawn in the canvas or whether the predicted drawing stroke is a movement of the dynamic drawing window to a different region of the input image without drawing the predicted drawing stroke in the canvas.

13.

Assignees

• Zou Changqing
• Wang Mingxue
• Arora Himanshu
• Huawei Tech Co Ltd

Inventors

• Zou Changqing
• Wang Mingxue
• Arora Himanshu

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T11/23 Primary

  using straight lines or curves · CPC title

• G06T11/203 Primary

  Physics · mapped topic

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06T11/001

  Physics · mapped topic