Systems and methods for inferring gender by fusion of multimodal content

Patent metadata

Field	Value
Publication number	US-9684852-B2
Application number	US-201615226789-A
Country	US
Kind code	B2
Filing date	Aug 2, 2016
Priority date	Jun 29, 2015
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

A method and systems are provided. A system includes a set of visual and textual classifiers for recognizing semantic concepts in a set of images and assigning semantic scores for the images to predict a gender of a user, and performing gender prediction from visual content and textual content in the images to respectively generate visual-based gender predictions and textual-based gender predictions. The system further includes a multimodal information fusion device for combining, using multimodal information fusion, the visual-based gender predictions, the textual-based gender predictions, and the semantic scores to infer a gender of a user.

First claim

What is claimed is:

1. A system, comprising:
a set of visual and textual classifiers for recognizing semantic concepts in a set of images and assigning semantic scores for the images to predict a gender of a user, and performing gender prediction from visual content and textual content in the images to respectively generate visual-based gender predictions and textual-based gender predictions; and
a multimodal information fusion device for combining, using multimodal information fusion, the visual-based gender predictions, the textual-based gender predictions, and the semantic scores to infer a gender of a user.

2. The system of claim 1 , wherein the semantic concepts comprise visual-based semantic concepts and textual-based semantic concepts.

3. The system of claim 1 , wherein at least one of the visual classifiers and at least one of the textual classifiers generate semantic distributions from the semantic scores, and wherein some of the visual-based gender predictions and the textual-based gender predictions are determined from the semantic distributions.

4. The system of claim 1 , wherein at least one of the visual classifiers and at least one of the textual classifiers generate semantic distributions from the semantic scores, wherein some of the visual-based gender predictions and the textual-based gender predictions are determined from aggregations of the semantic distributions.

5. The system of claim 1 , wherein said multimodal information fusion device performs filtered fusion by selecting which multimodal gender information to use directly and which of the multimodal gender information to aggregate, to infer the gender of the user, from among the visual-based gender predictions, the textual-based gender predictions, and the semantic scores.

6. The system of claim 1 , wherein said multimodal information fusion device further comprises a confidence checker for performing confidence checking on at least some of the semantic scores to influence an inference of the gender of the user.

7. The system of claim 6 , wherein the confidence checking is performed separately by a first confidence check on the at least some of the semantic scores relating to the visual content and by a second confidence check on the at least some of the semantic scores relating to the textual content, and the inference of the gender of the user is influenced differently based on an agreement or a non-agreement between the first and the second confidence checks.

8. The system of claim 1 , wherein the visual content comprises profile colors and a background scene in a profile picture of the user.

9. A system for inferring gender, comprising:
a set of visual classifiers, each for recognizing visual content from a set of images associated with a user, assigning respective visual-based prediction confidence scores for each of the images based the recognized visual-based content, and at least one for generating a visual-based gender prediction of the user based on the respective visual-based prediction confidence scores for each of the images;
a set of textual classifiers, each for recognizing textual content from the set of images, assigning respective textual-based prediction confidence scores for each of the images based the recognized textual content, and at least one for generating a visual-based gender prediction of the user based on the respective visual-based prediction confidence scores for each of the images; and
a multimodal information fusion device for inferring a gender of a user selectively from the visual-based gender predictions, the textual-based gender predictions, the visual-based prediction confidence scores and the textual-based prediction confidence scores.

10. The system of claim 9 , wherein the at least one of the visual classifiers determines respective semantic distributions across the set of images for different ones of the recognized visual content, each of the respective semantic distribution based on an aggregation of the respective visual-based prediction confidence scores for a same one of the recognized visual content for each of the images.

11. The system of claim 9 , wherein the at least one of the visual classifiers generates the visual-based gender prediction based on the respective semantic distributions.

Assignees

• IBM

Inventors

• Cao Liangliang
• Merler Michele
• Smith John R

Classifications

• G06F18/2431

  Multiple classes · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/04

  Inference or reasoning models · CPC title