Image scoring and identification based on facial feature descriptors

Patent metadata

Field	Value
Publication number	US-11430256-B2
Application number	US-202016813284-A
Country	US
Kind code	B2
Filing date	Mar 9, 2020
Priority date	Apr 29, 2014
Publication date	Aug 30, 2022
Grant date	Aug 30, 2022

Abstract

Technologies for grouping images, and ranking the images and the groupings, based on entities shown in the images. Images may be grouped based on faces shown in the images. Different images with faces that indicate the same entity (e.g., Adam) may be automatically grouped together. Different images with faces that indicate the same multiple entities (e.g., the people in my family) may also be automatically grouped together. Such automatic grouping may be based on facial recognition technologies. Further, images and groups of images may be automatically ranked based on the faces shown and entities represented. Such rankings may also be influenced by adjacent data that indicates family and friends and the like, and that can be used to identify such entities in the images.

First claim

The invention claimed is:

1. A method performed on a computing device comprising:
receiving, by the computing device, digital image comprised of pixels;
detecting, by the computing device, a face in the image based on values of pixels within the digital image and location coordinates of the pixels within the digital image;
generating, by the computing device, a set of facial feature descriptors including coordinates describing shapes of the face's eyes and mouth within the image, wherein individual facial feature descriptors describe individual features that differ from features described by other facial feature descriptors in the set of facial feature descriptors, the set of facial feature descriptors comprising multiple facial feature descriptors that describe multiple features of the detected face;
generating, by the computing device, a face score based on an open/closed state of the face's eyes and mouth and that indicates an overall quality of the face; and
assigning, by the computing device, a face signature that uniquely identifies an entity that the face represents using the set of facial feature descriptors.

2. The method of claim 1 , further comprising:
detecting a second face within the image;
generating a second set of facial feature descriptors including coordinates for the second face's eyes and mouth within the image;
generating a second face score based on an open/closed state of the second face's eyes and mouth and that indicates an overall quality of the second face; and
assigning a second face signature that uniquely identifies an entity that the face represents using the set of facial feature descriptors.

3. The method of claim 2 , wherein the face signature and the second face signature are different when the face and the second face depict different entities.

4. The method of claim 2 , wherein the face signature and the second face signature are the same when the face and the second face depict the same entity.

5. The method of claim 1 , further comprising:
assigning a unique face identifier to the detected face.

6. The method of claim 5 , wherein the face identifier includes one or both of a position and a relative size of the detected face in the image.

7. The method of claim 1 , further comprising:
outputting facial recognition data that includes one or more of the set of facial feature descriptors, the face score, the face signature, and the received image to one or more of a data store, a grouping engine, and a ranking engine.

8. The method of claim 1 , wherein the set of facial feature descriptors further includes coordinates for one or both of the face's eyebrows and nose within the image.

9. The method of claim 1 , wherein the face score is further based on one or more of face sharpness, face expression, face pose, and face proximity to an edge of the image.

10. A system including a computing device and a facial recognition engine that together are configured to perform actions comprising:
receiving digital image comprised of pixels;
detecting a face in the image based on values of pixels within the digital image and location coordinates of the pixels within the digital image;
generating a set of facial feature descriptors including coordinates describing shapes of the face's eyes and mouth within the image, wherein individual facial feature descriptors describe individual features that differ from features described by other facial feature descriptors in the set of facial feature descriptors, the set of facial feature descriptors comprising multiple facial feature descriptors that describe multiple features of the detected face;
generating a face score based on an open/closed state of the face's eyes and mouth and that indicates an overall quality of the face; and
assigning a face signature that uniquely identifies an entity that the face represents using the set of facial feature descriptors.

11. The system of claim 10 , further configured to perform an action comprising:
assigning a unique face identifier to the detected face.

12. The system of claim 10 , further configured to perform an action comprising:
outputting facial recognition data that includes one or more of the set of facial feature descriptors, the face score, the face signature, and the received image to one or more of a data store, a grouping engine, and a ranking engine.

13. A method performed on a computing device comprising:
receiving, by the computing device, a set of digital images, each digital image being comprised of pixels;
detecting, by the computing device, multiple faces within the set of images based on values of pixels within each digital image and location coordinates of the pixels within each digital image;
generating, by the computing device, a set of facial feature descriptors including coordinates describing shapes of eyes and mouth for each detected face within the set of images, wherein individual facial feature descriptors describe individual features that differ from features described by other facial feature descriptors in the set of facial feature descriptors, the set of facial feature descriptors comprising multiple facial feature descriptors that describe multiple features of each detected face;
generating, by the computing device, a face score based on an open/closed state of the eyes and mouth and that indicates an overall quality of each detected face;
assigning, by the computing device, a face signature to each detected face that uniquely identifies an entity that each detected face represents across the set of images using the set of facial feature descriptors.

14. The method of claim 13 , wherein the face signatures are different when depicting different entities and the face signatures are the same when depicting the same entity.

15. The method of claim 13 , further comprising:
assigning a unique face identifier to each detected face.

16. The method of claim 13 , further comprising:
outputting facial recognition data that includes one or more of the set of facial feature descriptors, the face score, and the face signature for each of the received images to one or more of a data store, a grouping engine, and a ranking engine.

17. The method of claim 13 , further comprising:
determining a frequency of occurrence of each assigned face signature within the set of images.

18. The method of claim 13 , further comprising:
grouping, based on the face signatures, at least a portion of the images in the set of images into a set of groups, each image within a group including a detected face that depicts the same entity.

19. The method of claim 18 , further comprising:
ranking, based on the face scores, images within each of the set of groups.

20. The method of claim 18 , further comprising:
ranking each of the set of groups based on a number of entities depicted in each group.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chan Chunkit Jacky
• Lee David
• Light Allison

Classifications

• G06V20/30 Primary

  in albums, collections or shared content, e.g. social network photos or video · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V40/16 Primary

  Human faces, e.g. facial parts, sketches or expressions · CPC title