Visual recognition using social links

Patent metadata

Field	Value
Publication number	US-10204090-B2
Application number	US-201715651822-A
Country	US
Kind code	B2
Filing date	Jul 17, 2017
Priority date	Mar 17, 2014
Publication date	Feb 12, 2019
Grant date	Feb 12, 2019

Abstract

System, method and architecture for providing improved visual recognition by modeling visual content, semantic content and an implicit social network representing individuals depicted in a collection of content, such as visual images, photographs, etc., which network may be determined based on co-occurrences of individuals represented by the content, and/or other data linking the individuals. In accordance with one or more embodiments, using images as an example, a relationship structure may comprise an implicit structure, or network, determined from co-occurrences of individuals in the images. A kernel jointly modeling content, semantic and social network information may be built and used in automatic image annotation and/or determination of relationships between individuals, for example.

First claim

The invention claimed is:

1. A method comprising:
training, by at least one computing device, a machine-modeled kernel jointly modeling content, semantic information and social network information, the training comprising building the kernel using a content kernel trained using a training set comprising content feature information, a semantic kernel trained using semantic feature information of the training data set and a social network kernel trained using social network feature information of the training data set, an implicit social network determined from a plurality of content items is used in determining the social network feature information used in training the kernel;
identifying, by the at least one computing device and using a number of content items other than the plurality of content items, a plurality of individuals depicted in the number of content items using the kernel;
identifying, by the at least one computing device, a number of relationships, each relationship being between two individuals, of the plurality of individuals, identified in a same content item, of the number of content items, using the kernel; and
representing, by the at least one computing device, each relationship, of the number of identified relationships, in an electronic social network comprising a plurality of nodes and a plurality of connections, each identified relationship being represented as a connection between a pair of individuals, of the plurality of individuals, identified in the same content item using the kernel and each individual of the pair being represented as a node of the plurality of nodes.

2. The method of claim 1 , further comprising identifying at least one community comprising a number of individuals, of the plurality of individuals, identified as being connected using the kernel.

3. The method of claim 2 , further comprising identifying a connection between at least two communities of individuals identified using the kernel, identification of the connection between the at least two communities being based on an occurrence of one or more individuals belonging to each community of the at least two communities.

4. The method of claim 2 , at least one interest being assigned to each community of the at least one community.

5. The method of claim 1 , further comprising determining, for the connection between the pair of individuals, a connection strength for the pair of individuals, determination of the connection strength being based on how many of the number of content items the pair of individuals is depicted in the same content item.

6. The method of claim 1 , further comprising
serving, by the at least one computing device and to a client computing device of a user via an electronic communications network, content, the serving of the content to the user computing device resulting in the content being output by the user computing device.

7. The method of claim 1 , the trained kernel modeling relationships among content, semantic and social network features.

8. The method of claim 1 , the content item's social network information identifying each individual represented in the content item.

9. The method of claim 1 , the content item's semantic information identifying one or more annotations associated with the content item.

10. The method of claim 1 , further comprising:
training, by at least one computing device, the content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;
training, by at least one computing device, the semantic kernel using at least the semantic feature information associated with each content item of the plurality of content items in the training set; and
training, by at least one computing device, the social kernel using at least the social network feature information associated with each content item of the plurality of content items in the training set.

11. The method of claim 1 , the kernel jointly modeling content, semantic and social network information comprising the content, semantic and social network kernels and a weighting for each of the content, semantic and social network kernels.

12. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:
training a machine-modeled kernel jointly modeling content, semantic information and social network information, the training comprising building the kernel using a content kernel trained using a training set comprising content feature information, a semantic kernel trained using semantic feature information of the training data set and a social network kernel trained using social network feature information of the training data set, an implicit social network determined from a plurality of content items is used in determining the social network feature information used in training the kernel;
identifying, using a number of content items other than the plurality of content items, a plurality of individuals depicted in the number of content items using the kernel;
identifying a number of relationships, each relationship being between two individuals, of the plurality of individuals, identified in a same content item, of the number of content items, using the kernel; and
representing each relationship, of the number of identified relationships, in an electronic social network comprising a plurality of nodes and a plurality of connections, each identified relationship being represented as a connection between a pair of individuals, of the plurality of individuals, identified in the same content item using the kernel and each individual of the pair being represented as a node of the plurality of nodes.

13. The non-transitory computer-readable storage medium of claim 12 , further comprising identifying at least one community comprising a number of individuals, of the plurality of individuals, identified as being connected using the kernel.

14. The non-transitory computer-readable storage medium of claim 13 , further comprising identifying a connection between at least two communities of individuals identified using the kernel, identification of the connection between the at least two communities being based on an occurrence of one or more individuals belonging to each community of the at least two communities.

15. The non-transitory computer-readable storage medium of claim 13 , at least one interest being assigned to each community of the at least one community.

16. The non-transitory computer-readable storage medium of claim 12 , further comprising determining, for the connection between the pair of individuals, a connection strength for the pair of individuals, determination of the connection strength being based on how many of the number of content items the pair of individuals is depicted in the same content item.

17. The non-transitory computer-readable storage medium of claim 12 , further comprising
serving, to a client computing device of a user via an electronic communications network, content, the serving of the content to the user computing device resulting in the content being output by the user computing device.

18. The non-transitory computer-readable storage medium of claim 12 , further comprising:
training the content kernel using at least the content feature information associated with each content item of the plurality of content items in the training set;
training the semantic kernel using at least the semantic feature inf

Assignees

• Oath Inc

Inventors

• Li Jia
• Kong Xiangnan

Classifications

• G06V10/84

  using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06K9/00677

  Physics · mapped topic