Providing drawing assistance using feature detection and semantic labeling

Patent metadata

Field	Value
Publication number	US-9495581-B2
Application number	US-201414176034-A
Country	US
Kind code	B2
Filing date	Feb 7, 2014
Priority date	Feb 3, 2014
Publication date	Nov 15, 2016
Grant date	Nov 15, 2016

Abstract

Methods for providing drawing assistance to a user sketching an image include geometrically correcting adjusting user strokes to improve their placement and appearance. In particular, one or more guidance maps indicate where the user “should” draw lines. As a user draws a stroke, the stroke is geometrically corrected by moving the stroke toward a portion of the guidance maps corresponding to the feature of the image the user is intending to draw based feature labels. To further improve the user drawn lines, parametric adjustments are optionally made to the geometrically-corrected stroke to emphasize “correctly” drawn lines and de-emphasize “incorrectly” drawn lines.

First claim

We claim:

1. A method of using semantic labeling to provide drawing assistance to a user sketching an image comprising:
generating, by one or more processors of a computing device, a guidance map for the image, the guidance map including feature labels identifying one or more portions of the guidance map corresponding to one or more features of the image;
detecting a stroke drawn by a user on a touch screen of the computing device;
associating a feature label with the stroke;
identifying a portion of the guidance map nearest to the stroke that has an associated feature label that matches the feature label of the stroke;
generating, by the one or more processors, a geometrically-corrected stroke using the identified portion of the guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke; and
rendering, by the computing device, the geometrically-corrected stroke on the touch screen in place of the stroke drawn by the user.

2. The method as recited in claim 1 , wherein generating the guidance map comprises generating an edge guidance map by applying an edge detection filter to the image.

3. The method as recited in claim 2 , wherein generating the guidance map further comprises:
applying a feature detection algorithm to the image to identify one or more features in the image; and
associating feature labels with edges in the edge guidance map corresponding to the identified features.

4. The method as recited in claim 2 , wherein generating the guidance map further comprises generating a feature guidance map by applying a feature detection algorithm to the image to identify one or more features in the image.

5. The method as recited in claim 4 , further comprising using the feature guidance map to supplement the edge guidance map by using feature lines of the feature guidance map for missing portions of the edge guidance map.

6. The method as recited in claim 4 , wherein generating the geometrically-corrected stroke using the identified portion of the guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke comprises:
determining a feature line of the feature guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke;
determining an edge of the edge guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke;
determining which of the edge or the feature line provides a better guide for the stroke; and
moving the stroke toward the determined edge or feature line that provides the better guide for the stroke.

7. The method as recited in claim 2 , wherein the edge detection filter comprises an extended Difference-of-Gaussians filter.

8. The method as recited in claim 1 , wherein generating the guidance map comprises generating a feature guidance map by applying a feature detection algorithm to the image to identify one or more features in the image.

9. The method as recited in claim 7 , wherein:
the image comprises a photograph of a face; and
applying a feature detection algorithm to the image comprises applying a face detection algorithm to the image.

10. The method as recited in claim 1 , further comprising determining deviations of the geometrically-corrected stroke from the identified portion of the guidance map by determining differences in position and curvature between the geometrically-corrected stroke and the identified portion of the guidance map.

11. The method as recited in claim 9 , further comprising one or more of:
de-emphasizing portions of the geometrically-corrected stroke that deviate from the identified portion of the guidance map; or
emphasizing portions of the geometrically-corrected stroke that align with the identified portion of the guidance map.

12. The method as recited in claim 1 , further comprising rendering the geometrically-corrected stroke as the user draws the stroke.

13. The method as recited in claim 1 , further comprising:
determining a variation of the stroke from the identified edge of the guidance map; and
at least partially maintaining the variation when geometrically correcting the stroke.

14. A system comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to:
generate a guidance map for the image, the guidance map including feature labels identifying one or more portions of the guidance map corresponding to features of the image;
detect a stroke drawn by a user on a touch screen;
associate a feature label with the stroke;
identify a portion of the guidance map nearest to the stroke that has an associated feature label that matches the feature label of the stroke;
generate a geometrically-corrected stroke using the stroke toward the identified portion of the guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke; and
render the geometrically-corrected stroke on the touch screen in place of the stroke drawn by the user.

15. The system as recited in claim 14 , wherein the instructions, when executed by the at least one processor, cause the system to generate the guidance map for the image by generating one or more of:
a feature guidance map of the image by applying a feature detection algorithm to the image that identifies positions of one or more features of the image; or
an edge guidance map of the image by applying an edge detection filter to the image.

16. The system as recited in claim 15 , wherein the identified portion of the guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke comprises one or more of:
a line of the feature guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke; or
an edge of the edge guidance map nearest to the stroke that has the associated feature label that matches the feature label of the stroke.

17. The system as recited in claim 15 , wherein the instructions, when executed by the at least one processor, further cause the system to:
determine a first distance of the stroke from the line of the feature guidance map;
determine a second distance of the stroke from the edge of the guidance map; and
give weight to the line of the feature guidance map and the edge of the edge guidance map based on the determined first and second distances when generating the geometrically-corrected stroke.

18. A computer-implemented method of using semantic labeling to provide drawing assistance to a user sketching an image comprising:
detecting one or more features in the image;
associating feature labels with guidance points in a guidance map corresponding to the detected features;
identifying input points defining a stroke drawn by the user on a touch screen;
associating feature labels with the input points of the stroke;
identifying guidance points of the guidance map nearest to the input points of the stroke that have associated feature labels that match the feature labels of the input points of the stroke;
identifying feature points nearest to the input points of the stroke that have associated feature labels that match the feature labels of the input points of the stroke;
determining geometrically-corrected points for the input points based on the identified guidance points and the identifie

Assignees

• Adobe Systems Inc

Inventors

• Winnemoeller Holger
• Xie Jun
• Li Wilmot Wei-Mau
• Hertzmann Aaron Phillip

Classifications

• G06T11/23

  using straight lines or curves · CPC title

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T11/203

  Physics · mapped topic

• G06K9/00281 Primary

  Physics · mapped topic

• G06T11/001

  Physics · mapped topic