Relationship mapping employing multi-dimensional context including facial recognition

Patent metadata

Field	Value
Publication number	US-9984098-B2
Application number	US-201514663166-A
Country	US
Kind code	B2
Filing date	Mar 19, 2015
Priority date	Mar 20, 2008
Publication date	May 29, 2018
Grant date	May 29, 2018

Abstract

A system and method for mapping interpersonal relationships, the method including processing a multiplicity of images and contextual information relating thereto including creating and prioritizing a list of a plurality of candidate persons having at least a predetermined relationship with at least one person connected to at least one image, using multi-dimensional information including visually sensible information in the multiplicity of images and contextual information relating thereto and searching the list of a plurality of candidate persons based at least in part on the prioritizing to select at least one of the candidate persons as having at least a predetermined relationship with the at least one person.

First claim

The invention claimed is:

1. A method comprising:
processing a first composite image that includes an image of at least one unknown person to create a list of candidate persons, each candidate person having a predetermined relationship with a specified degree of separation on an online social network with one or more persons connected to the first composite image, the processing being based on a facial-recognition comparison of a face representation of the unknown person with face representations of the candidate persons;
sending, to a client device of a first user, the list of candidate persons for display to the first user, the candidate persons being listed in an order prioritized based on the facial-recognition comparison and a measure of quality of the facial representation of the respective candidate person, each candidate person corresponding to a particular person identifier; and
receiving, from the client device of the first user, a selection of one of the candidate persons for tagging the image of the unknown person with the particular person identifier corresponding to the selected candidate person.

2. The method of claim 1 , further comprising tagging, responsive to the selection, the image of the unknown person with the particular person identifier corresponding to the selected candidate person.

3. The method of claim 1 , further comprising receiving, from the client device of the first user, the first composite image.

4. The method of claim 1 , wherein the processing of the first composite image is further based on processing of contextual information relating to the first composite image.

5. The method of claim 1 , further comprising accessing one or more second images and contextual information relating thereto, and wherein the processing of the first composite image is further based on processing of the one or more second images and the contextual information relating thereto.

6. The method of claim 5 , further comprising generating one or more face representations of one or more candidate persons based on the processing of the one or more second images.

7. The method of claim 1 , further comprising accessing multi-dimensional information relating to the first composite image, the multi-dimensional information comprising:
one or more of:
visual information appearing in the first composite image;
geographical information appearing in the first composite image;
visual background information appearing in the first composite image;
images of other persons appearing in the first composite image; or
person identifiers appearing in the first composite image; and

one or more of:
information relating to an image collection of which the first composite image forms a part;
a time stamp associated with the first composite image;
information not appearing in the first composite image but associated therewith;
geographical information not appearing in the first composite image;
visual background information appearing in one or more second images; or
person identifiers not appearing in the first composite image but otherwise associated therewith.

8. The method of claim 1 , wherein the first composite image includes images of multiple persons, none of whom are known.

9. The method of claim 1 , wherein one or more of the persons connected to the first composite image appears in the first composite image.

10. The method of claim 1 , wherein the processing of the first composite image comprises iteratively generating relationship maps based at least on visually sensible information and non-visually sensible information related to persons who appear in the first composite image or are otherwise associated with the first composite image.

11. The method of claim 10 , wherein iteratively generating relationship maps is further based on social relationship data accessed from an online social network.

12. The method of claim 10 , wherein the non-visually sensible information comprises meta-data associated with image data of the first composite image.

13. The method of claim 1 , wherein the processing of the first composite image comprises prioritizing the list of candidate persons based on an indication of whether a candidate person appears in the same album in an online social network as the first composite image.

14. The method of claim 13 , wherein the facial-recognition comparison is processed in accordance with and in an order established by the prioritizing.

15. The method of claim 1 , further comprising identifying one or more candidate persons, each candidate person having at least a predetermined relationship with a known person in the first composite image.

16. The method of claim 1 , wherein identifying the one or more candidate persons comprises a comparison of a face representation of each candidate person with a face representation of one or more persons in a relationship map associated with the first composite image.

17. The method of claim 1 , wherein the list of candidate persons is displayed to the first user on a user interface of a native application associated with an online social network on the client device of the first user.

18. The method of claim 1 , wherein the list of candidate persons is displayed to the first user on a webpage of an online social network accessed by a browser client on the client device of the first user.

19. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
process a first composite image that includes an image of at least one unknown person to create a list of candidate persons, each candidate person having a predetermined relationship with a specified degree of separation on an online social network with one or more persons connected to the first composite image, the processing being based on a facial-recognition comparison of a face representation of the unknown person with face representations of the candidate persons;
send, to a client device of a first user, the list of candidate persons for display to the first user the candidate persons being listed in an order prioritized based on the facial-recognition comparison and a measure of quality of the facial representation of the respective candidate person, each candidate person corresponding to a particular person identifier; and
receive, from the client device of the first user, a selection of one of the candidate persons for tagging the image of the unknown person with the particular person identifier corresponding to the selected candidate person.

20. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
process a first composite image that includes an image of at least one unknown person to create a list of candidate persons, each candidate person having a predetermined relationship with a specified degree of separation on an online social network with one or more persons connected to the first composite image, the processing being based on a facial-recognition comparison of a face representation of the unknown person with face representations of the candidate persons;
send, to a client device of a first user, the list of candidate persons for display to the first user the candidate persons being listed in an order prioritized based on the facial-recognition comparison and a measure of quality of the facial representation of the respective candidate person, each candidate person corresponding to a

Assignees

• Facebook Inc

Inventors

• Shochat Eden
• Shniberg Moti
• Hirsch Gil
• Taigman Yaniv

Classifications

• G06V10/768

  using context analysis, e.g. recognition aided by known co-occurring patterns · CPC title

• G06F16/5838 Primary

  using colour · CPC title

• G06F16/5866 Primary

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title

• G06F17/30256 Primary

  Physics · mapped topic

• G06F17/30241

  Physics · mapped topic