Converting a text sentence to a series of images

Patent metadata

Field	Value
Publication number	US-9633048-B1
Application number	US-201514941955-A
Country	US
Kind code	B1
Filing date	Nov 16, 2015
Priority date	Nov 16, 2015
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

A text sentence is automatically converted to an image sentence that conveys semantic roles of the text sentence. This is accomplished by identifying semantic roles associated with each verb of a sentence, any associated verb adjunctions, and identifying the grammatical dependencies between words and phrases in a sentence, in some embodiments. An image database, in which each image is tagged with descriptive information corresponding to the image depicted, is queried for images corresponding to the semantic roles of the identified verbs. Unless a single image is found to depict every semantic role, the text sentence is split into two smaller fragments. This process is the repeated and performed recursively until a number of images have been identified that describe each semantic role of each sentence fragment.

First claim

What is claimed is:

1. A method for converting a text sentence into an image sentence, the method comprising:
receiving a text sentence comprising a plurality of text words, the plurality of text words including at least one verb phrase;
identifying at least one semantic role associated with the at least one verb phrase;
querying an image database for a single image associated with information matching at least one semantic role, the querying including generating a first feature vector of first candidate images in the image database and comparing the first feature vector to the at least one semantic role associated with the at least one verb phrase;
responsive to determining that no single image is associated with information matching the semantic role, splitting the text sentence into a first sentence fragment and a second sentence fragment, each of the first sentence fragment and the second sentence fragment associated with at least one semantic role; and
querying the image database for a first image associated with information matching the at least one semantic role of the first sentence fragment and a second image associated with information matching the at least one semantic role of the second sentence fragment, the querying including generating a second feature vector for second candidate images and comparing the second feature vector to the at least one semantic roles of the first sentence fragment and the second sentence fragment.

2. The method of claim 1 , wherein at least one of the first sentence fragment and the second sentence fragment includes an adjunct associated with at least one verb phrase.

3. The method of claim 1 , further comprising presenting the first image and the second image as an image sentence corresponding to the text sentence.

4. The method of claim 1 , further comprising:
determining whether the first image matches all of the at least one semantic roles of the first sentence fragment and whether the second image matches all of the at least one semantic roles of the second sentence fragment.

5. The method of claim 4 , further comprising:
responsive to determining that at least one of the first image and the second image do not correspond to all semantic roles of the first sentence fragment and the second sentence fragment, splitting at least one of the first sentence fragment and the second sentence fragment into two more sentence fragments.

6. The method of claim 1 , further comprising:
determining that the first image matches all semantic roles of the first sentence fragment and that the second image matches all semantic roles of the second sentence fragment; and
presenting the first image and the second image as an image sentence corresponding to the text sentence.

7. A computer program product wherein the computer program product is stored on at least one non-transitory computer-readable medium that includes instructions that when executed by one or more processors cause a process to be carried out, the process comprising:
receiving a text sentence comprising a plurality of text words, the plurality of text words including at least one verb phrase;
identifying at least one semantic role associated with the at least one verb phrase;
querying an image database for a single image associated with information matching at least one semantic role, the querying including generating a first feature vector of first candidate images in the image database and comparing the first feature vector to the at least one semantic role associated with the at least one verb phrase;
responsive to determining that no single image is associated with information matching the semantic role, splitting the text sentence into a first sentence fragment and a second sentence fragment, each of the first and second sentence fragments associated with one of the at least one semantic roles; and
querying the image database for a first image associated with information matching the semantic role of the first sentence fragment and a second image associated with information matching the semantic role of the second sentence fragment, the querying including generating a second feature vector for second candidate images and comparing the second feature vector to the at least one semantic roles of the first sentence fragment and the second sentence fragment.

8. The computer program product of claim 7 , wherein at least one of the first sentence fragment and the second sentence fragment includes an adjunct associated with at least one verb phrase.

9. The computer program product of claim 7 , further comprising presenting the first image and the second image as an image sentence corresponding to the text sentence.

10. The computer program product of claim 7 , further comprising:
determining whether the first image matches all of the at least one semantic roles of the first sentence fragment and whether the second image matches all of the at least one semantic roles of the second sentence fragment.

11. The computer program product of claim 10 , further comprising:
responsive to determining that at least one of the first image and the second image do not correspond to all semantic roles of the first sentence fragment and the second sentence fragment, splitting at least one of the first sentence fragment and the second sentence fragment into two more sentence fragments.

12. The computer program product of claim 7 , further comprising:
determining that the first image matches all semantic roles of the first sentence fragment and that the second image matches all semantic roles of the second sentence fragment; and
presenting the first image and the second image as an image sentence corresponding to the text sentence.

13. A system for converting a text sentence into an image sentence, the system comprising:
at least one processor;
a text sentence analyzer configured for:
receiving a text sentence comprising a plurality of text words, the plurality of text words including at least one verb phrase;
identifying at least one semantic role associated with the at least one verb phrase;
splitting the text sentence into a first sentence fragment and a second sentence fragment, each of the first sentence fragment and the second sentence fragment associated with at least one semantic role;

an image database configured for storing images and corresponding information describing semantic roles of the stored images;
a text/image comparison module configured for:
receiving the text sentence;
determining at least one semantic role associated with the at least one verb phrase;
querying the image database for a single image associated with information matching the semantic roles of the at least one verb phrase, the querying including generating a feature vector of a candidate image in the image database and comparing the feature vector of the candidate image to the semantic roles associated with the at least one verb phrase; and
responsive to determining that no single image is associated with information matching the semantic roles of the at least one verb phrase, instructing the text sentence analyzer to split the text sentence into the first sentence fragment and the second sentence fragment.

14. The system of claim 13 , wherein at least one of the first sentence fragment and the second sentence fragment is an adjunct associated with the at least one verb phrase.

15. The system of claim 13 , wherein the text/image comparison module is further configured for:
receiving the first sentence fragment and the second sentence fragment from the text sentence analyzer; and
querying the image database for a first imag

Assignees

• Adobe Systems Inc

Inventors

• Dutta Siddhartha Kumar
• Gundapuneni Midhun
• Jhamtani Harsh
• Varma Shubham

Classifications

• G06F40/211

  Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F40/131

  Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces · CPC title