Image cropping suggestion using multiple saliency maps

Patent metadata

Field	Value
Publication number	US-9626584-B2
Application number	US-201414511001-A
Country	US
Kind code	B2
Filing date	Oct 9, 2014
Priority date	Oct 9, 2014
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

Image cropping suggestion using multiple saliency maps is described. In one or more implementations, component scores, indicative of visual characteristics established for visually-pleasing croppings, are computed for candidate image croppings using multiple different saliency maps. The visual characteristics on which a candidate image cropping is scored may be indicative of its composition quality, an extent to which it preserves content appearing in the scene, and a simplicity of its boundary. Based on the component scores, the croppings may be ranked with regard to each of the visual characteristics. The rankings may be used to cluster the candidate croppings into groups of similar croppings, such that croppings in a group are different by less than a threshold amount and croppings in different groups are different by at least the threshold amount. Based on the clustering, croppings may then be chosen, e.g., to present them to a user for selection.

First claim

What is claimed is:

1. A method implemented by a computing device, the method comprising:
generating multiple different types of saliency maps of a scene, including a base saliency map computed from color model information describing the scene and at least one other type of saliency map;
computing component scores for candidate image croppings of the scene based on the base saliency map and the at least one other type of saliency map, the component scores indicative of visual characteristics established for visually pleasing croppings; and
choosing one or more image croppings from the candidate image croppings using the component scores.

2. A method as described in claim 1 , wherein each of the component scores is indicative of a different visual characteristic.

3. A method as described in claim 1 , wherein the at least one other type of saliency map includes at least one of a dense saliency map, a saliency edge map, a row-normalized gradient map, or an image border saliency map.

4. A method as described in claim 1 , wherein at least one of the component scores indicates a composition quality of the candidate image croppings, the composition quality determined based on a comparison of the candidate image croppings to composition properties derived from well-composed images.

5. A method as described in claim 1 , wherein at least one of the component scores is indicative of whether at least a portion of content appearing in the scene is preserved by the candidate image croppings.

6. A method as described in claim 1 , wherein at least one of the component scores is indicative of a simplicity of a boundary of the candidate image croppings.

7. A method as described in claim 1 , wherein at least one of the component scores indicates a degree to which the candidate image croppings preserve specified regions-to-keep and exclude specified regions-to-remove.

8. A method as described in claim 7 , wherein at least one of the regions-to-keep or the regions-to-remove corresponds to an object that is automatically detected in the image without user interaction.

9. A method as described in claim 7 , wherein at least one of the regions-to-keep or the regions-to-remove is specified according to user input to keep or remove the at least one region-to-keep or region-to-remove.

10. A method as described in claim 1 , further comprising:
ranking the candidate image croppings with regard to each of the visual characteristics based on corresponding component scores;
calculating an average ranking for each of the candidate image croppings by combining the rankings of a given candidate image cropping for each of the visual characteristics; and
using the average rankings to cluster the candidate image croppings into clusters of similar image croppings.

11. A method as described in claim 10 , wherein choosing the one or more image croppings from the candidate image croppings includes choosing the candidate image croppings that are ranked highest in a respective cluster.

12. A method as described in claim 10 , wherein choosing the one or more image croppings from the candidate image croppings includes choosing the candidate image croppings from different clusters.

13. A method as described in claim 10 , wherein the candidate image croppings in one cluster are different from the candidate image croppings in another cluster by at least a threshold amount.

14. A method as described in claim 10 , wherein the candidate image croppings in a given cluster are different, one candidate image cropping from another, by less than a threshold amount.

15. A system comprising:
one or more processors; and
a memory having stored thereon computer-readable instructions that are executable by the one or more processors to implement an image cropping module to:
compute component scores for candidate image croppings of a scene, the component scores indicative of visual characteristics established for visually pleasing croppings and computed using multiple different saliency maps;
rank the candidate image croppings with regard to each of the visual characteristics based on corresponding component scores;
calculate an average ranking for each of the candidate image croppings by combining the rankings of a given candidate image cropping for each of the visual characteristics;
cluster the candidate image croppings based on the average rankings into clusters of similar image croppings; and
choose one or more image croppings from the candidate image croppings based on the clusters.

16. A system as described in claim 15 , wherein each of the component scores is indicative of a different visual characteristic.

17. A system as described in claim 15 , wherein each of the component scores is computed using multiple different saliency maps.

18. A system as described in claim 15 , wherein at least one of the component scores indicates a composition quality of the candidate image croppings, the composition quality determined based on a comparison of the candidate image croppings to composition properties derived from well-composed images.

19. A system as described in claim 15 , wherein at least one of the component scores is indicative of whether at least a portion of content appearing in the scene is preserved by the candidate image croppings.

20. A computer-readable storage memory comprising an image editing application stored as instructions that are executable and, responsive to execution of the instructions by a computing device, the computing device performs operations of the image editing application comprising to:
compute component scores for candidate image croppings of a scene, the component scores indicative of visual characteristics established for visually pleasing croppings and computed using multiple different saliency maps;
rank the candidate image croppings with regard to each of the visual characteristics based on corresponding component scores;
calculate an average ranking for each of the candidate image croppings by combining the rankings of a given candidate image cropping for each of the visual characteristics;
cluster the candidate image croppings based on the average rankings into clusters of similar image croppings; and
choose one or more image croppings from at least two different clusters.

Assignees

• Adobe Systems Inc

Inventors

• Lin Zhe
• Mech Radomir
• Shen Xiaohui
• Price Brian L
• Zhang Jianming
• Gilra Anant
• Chien Jen-Chan Jeff

Classifications

• G06V10/462 Primary

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06K9/4671 Primary

  Physics · mapped topic

• G06T3/0012

  Physics · mapped topic

• G06T2210/22

  Cropping · CPC title