Intelligent auto cropping of digital images

Patent metadata

Field	Value
Publication number	US-10318794-B2
Application number	US-201715582087-A
Country	US
Kind code	B2
Filing date	Apr 28, 2017
Priority date	Apr 28, 2017
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Disclosed in some examples are methods, systems, computing devices, and machine readable mediums that provide for cropping systems that automatically crop digital images using one or more smart cropping techniques. Smart cropping techniques may include: cropping an image based upon emotion detection, cropping based upon facial recognition and matching, and cropping based upon landmark matching. In some examples, a single smart cropping technique may be utilized. In other examples, a combination of the smart cropping techniques may be utilized.

First claim

What is claimed is:

1. A machine-readable storage device, storing instructions, which when executed by a machine, cause the machine to perform operations comprising:
receiving a digital image;
detecting three or more faces in the digital image;
for each particular face in the three or more faces, determining an emotion displayed by the particular face;
clustering the three or more faces into two or more clusters, each particular cluster comprising faces displaying emotions that are the same or are classified as related to other faces in the particular cluster;
selecting a cluster of the two or more clusters; and
cropping the digital image based upon the faces in the selected cluster.

2. The machine-readable storage device of claim 1 , wherein the operations of cropping the digital image based upon the faces in the selected cluster comprises operations of cropping the digital image so as not to include a face in a cluster of the two or more clusters that was not selected.

3. The machine-readable storage device of claim 1 , wherein the operations further comprise:
for each particular face in the set of three or more faces, searching for a matching face in an image storage corresponding to a user; and
responsive to determining that a face in the set of three or more faces matches a face in an image stored in the image storage of the user, cropping the digital image based additionally upon the matched face.

4. The machine-readable storage device of claim 3 , wherein the image storage is a network-based image storage.

5. The machine-readable storage device of claim 3 , wherein the image storage is a social networking service.

6. The machine-readable storage device of claim 5 , wherein images of the image storage correspond to social media posts of the user or posts of connections of the user on the social networking service.

7. The machine-readable storage device of claim 1 , wherein the operations further comprise:
detecting a landmark in the digital image; and
cropping the digital image based additionally upon the landmark.

8. The machine-readable storage device of claim 7 , wherein the landmark is detected based upon a comparison of features of the digital image with features of a library of digital images including landmarks.

9. The machine-readable storage device of claim 1 , wherein the operations further comprise:
for each particular face in the set of three or more faces, searching for a matching face in an image storage of a user;
detecting a landmark in the digital image; and
cropping the digital image based upon the matching face, the landmark, and the set of faces in the selected cluster.

10. The machine-readable storage device of claim 9 , wherein the operations of cropping the digital image based upon the matching face, the landmark, and the faces in the selected cluster comprises the operations of:
producing a first score based upon a first confidence value corresponding to the set of three or more faces in the selected cluster and a first weighting factor;
producing a second score based upon a second confidence value corresponding to the matching face and a second weighting factor;
producing a third score based upon a third confidence value corresponding to the landmark and a third weighting factor; and
cropping the digital image based upon the first, second and third scores.

11. The machine-readable storage device of claim 10 , wherein the first, second, and third weighting factors are determined based upon a machine learning algorithm.

12. The machine-readable storage device of claim 11 , wherein a rejection by the user of the cropped digital image is used to adjust one or more of the first, second, or third weighting factors.

13. A system for cropping an image, the system comprising:
a processor;
a memory including instructions, which when executed by the processor, cause the system to perform operations comprising:
receiving a digital image;
detecting three or more faces in the digital image;
for each particular face in the three or more faces, determining an emotion displayed by the particular face;
clustering the three or more faces into two or more clusters, each particular cluster comprising faces displaying emotions that are the same or are classified as related to other faces in the particular cluster;
selecting a cluster of the two or more clusters; and
cropping the digital image based upon the faces in the selected cluster.

14. The system of claim 13 , wherein the operations of cropping the digital image based upon the faces in the selected cluster comprises operations of cropping the digital image so as not to include a face in a cluster of the two or more clusters that was not selected.

15. The system of claim 13 , wherein the operations further comprise:
for each particular face in the set of three or more faces, searching for a matching face in an image storage corresponding to a user; and
responsive to determining that a face in the set of three or more faces matches a face in an image stored in the image storage of the user, cropping the digital image based additionally upon the matched face.

16. The system of claim 13 , wherein the operations further comprise:
for each particular face in the set of three or more faces, searching for a matching face in an image storage of a user;
detecting a landmark in the digital image; and
cropping the digital image based upon the matching face, the landmark, and the set of faces in the selected cluster.

17. A method for cropping an image, the method comprising:
receiving a digital image;
detecting three or more faces in the digital image;
for each particular face in the three or more faces, determining an emotion displayed by the particular face;
clustering the three or more faces into two or more clusters, each particular cluster comprising faces displaying emotions that are the same or are classified as related to other faces in the particular cluster;
selecting a cluster of the two or more clusters; and
cropping the digital image based upon the faces in the selected cluster.

18. The method of claim 17 , comprising:
for each particular face in the set of three or more faces, searching for a matching face in an image storage corresponding to a user; and
responsive to determining that a face in the set of three or more faces matches a face in an image stored in the image storage of the user, cropping the digital image based additionally upon the matched face.

19. The method of claim 17 , comprising:
for each particular face in the set of three or more faces, searching for a matching face in an image storage of a user;
detecting a landmark in the digital image; and
cropping the digital image based upon the matching face, the landmark, and the set of faces in the selected cluster.

20. The method of claim 19 , wherein cropping the digital image based upon the matching face, the landmark, and the faces in the selected cluster comprises:
producing a first score based upon a first confidence value corresponding to the set of three or more faces in the selected cluster and a first weighting factor;
producing a second score based upon a second confidence value corresponding to the matching face and a second weighting factor;
producing a third score based upon a third confidence value corresponding to the landmark and a third weighting factor; and
cropping the digital image based upon the first, second and third scores.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lee David Benjamin
• Kikin Gil Erez

Classifications

• G06V40/175 Primary

  Static expression · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2210/22

  Cropping · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title