Determining contextual confidence of images using associative deep learning

Patent metadata

Field	Value
Publication number	US-2020082002-A1
Application number	US-201816129095-A
Country	US
Kind code	A1
Filing date	Sep 12, 2018
Priority date	Sep 12, 2018
Publication date	Mar 12, 2020
Grant date	—

Abstract

Determining contextual confidence of images for associative deep learning includes receiving an image including a representation of a subject. Text data related to the image is received. One or more physical properties of the image are determined. Context information of the image is determined using natural language processing. The image is classified based upon the contextual information and the one or more physical properties using a classification model to determine a classification. An emotional state of the image is determined based upon the physical properties. A confidence of the classification and emotional state is determined.

First claim

What is claimed is:

1 . A method for determining contextual confidence of images for associative deep learning, the method comprising:
receiving an image including a representation of a subject; receiving text data related to the image; determining one or more physical properties of the image; determining context information of the image using natural language processing; classifying the image based upon the contextual information and the one or more physical properties using a classification model to determine a classification; determining an emotional state of the image based upon the physical properties; and determining a confidence of the classification and emotional state.

2 . The method of claim 1 , further comprising:
updating the classification model based upon the determined confidence.

3 . The method of claim 1 , wherein the classification model includes a deep neural network.

4 . The method of claim 1 , wherein the confidence of the classification and emotional state is determined using negative class training.

5 . The method of claim 4 , wherein the negative class training includes using a probability distribution between a set of images with a known emotional state and a candidate image.

6 . The method of claim 5 , wherein the probability distribution is a Skellam distribution.

7 . The method of claim 5 , wherein the negative class training includes use of a Bessel function.

8 . The method of claim 1 , wherein the text data is included in a social media post associated with the image.

9 . The method of claim 1 , wherein the one or more physical properties of the image include one or more of a facial expression of the subject in the image, background of the image, or a relationship of objects in the image.

10 . The method of claim 1 , wherein the emotional state includes a mood of the subject.

11 . A computer usable program product comprising one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices, the stored program instructions comprising:
program instructions to receive an image including a representation of a subject; program instructions to receive text data related to the image; program instructions to determine one or more physical properties of the image; program instructions to determine context information of the image using natural language processing; program instructions to classify the image based upon the contextual information and the one or more physical properties using a classification model to determine a classification; program instructions to determine an emotional state of the image based upon the physical properties; and program instructions to determine a confidence of the classification and emotional state.

12 . The computer usable program product of claim 11 , further comprising:
program instructions to update the classification model based upon the determined confidence.

13 . The computer usable program product of claim 11 , wherein the classification model includes a deep neural network.

14 . The computer usable program product of claim 11 , wherein the confidence of the classification and emotional state is determined using negative class training.

15 . The computer usable program product of claim 14 , wherein the negative class training includes using a probability distribution between a set of images with a known emotional state and a candidate image.

16 . The computer usable program product of claim 15 , wherein the probability distribution is a Skellam distribution.

17 . The computer usable program product of claim 15 , wherein the negative class training includes use of a Bessel function.

18 . The computer usable program product of claim 11 , wherein the computer usable code is stored in a computer readable storage device in a data processing system, and wherein the computer usable code is transferred over a network from a remote data processing system.

19 . The computer usable program product of claim 11 , wherein the computer usable code is stored in a computer readable storage device in a server data processing system, and wherein the computer usable code is downloaded over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system.

20 . A computer system comprising one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions comprising:
program instructions to receive an image including a representation of a subject; program instructions to receive text data related to the image; program instructions to determine one or more physical properties of the image; program instructions to determine context information of the image using natural language processing; program instructions to classify the image based upon the contextual information and the one or more physical properties using a classification model to determine a classification; program instructions to determine an emotional state of the image based upon the physical properties; and program instructions to determine a confidence of the classification and emotional state.

Assignees

• IBM

Inventors

• Whitman Todd R
• Baughman Aaron K
• Perrino John P
• Pandey Diwesh

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/82

  using neural networks · CPC title