Multimedia document summarization

Patent metadata

Field	Value
Publication number	US-10762283-B2
Application number	US-201514947964-A
Country	US
Kind code	B2
Filing date	Nov 20, 2015
Priority date	Nov 20, 2015
Publication date	Sep 1, 2020
Grant date	Sep 1, 2020

Abstract

Multimedia document summarization techniques are described. That is, given a document that includes text and a set of images, various implementations generate a summary by extracting relevant text segments in the document and relevant segments of images with constraints on the amount of text and number/size of images in the summary.

First claim

What is claimed is:

1. In a digital medium environment including one or more computing devices configured to perform document summarization, a method comprising:
receiving a multimedia document to be processed to generate a summary;
receiving a set of parameters associated with a budget for processing the multimedia document, the set of parameters associated with the budget placing a constraint on a size of the summary and defining how many of a first content type and a second content type the summary is permitted to contain, wherein the first content type and the second content type are two different content types;
responsive to a determination of the constraint on the size of the summary specified by the budget having not been met, generating the summary for the multimedia document by:
determining that individual elements of the first content type and the second content type in the multimedia document are available within the budget to be added to the summary;
executing an objective function for each available element that can be added to the summary within the budget, the objective function providing a measure of a contribution of each available element to coverage of information content relative to content of the multimedia document, diversity of information content relative to content of the multimedia document, and cohesion of information content relative to content of the summary;
selecting an available element which maximizes a ratio of increase in the objective function to a cost of the element, wherein the cost of the element is a measure of the size of the element; and
adding the selected element to the summary.

2. The method as described in claim 1 , wherein receiving the set of parameters comprises receiving user-specified parameters.

3. The method as described in claim 1 , wherein receiving the set of parameters comprises receiving parameters associated with the first content type and the second content type.

4. The method as described in claim 1 , wherein the first content type comprises text and the second content type comprises images, and wherein receiving the set of parameters comprises receiving one or more of: number of sentences the summary is to have or number of images the summary is to have.

5. The method as described in claim 1 , wherein executing the objective function comprises iteratively executing the objective function to provide the summary.

6. The method as described in claim 1 , further comprising after receiving the multimedia document, assigning gain values and cost to each available element of the first content type and the second content type in the multimedia document, wherein the gain values are a function of the coverage that elements have regarding the information content of the document and diversity of information that the elements have with respect to the summary, and a higher gain value of an element results in a higher ratio of increase in the objective function to a cost of the element.

7. The method as described in claim 6 , further comprising:
updating gains of elements not in the summary;
updating a residual budget for corresponding element categories; and
iterating through, until the budget is exhausted: said executing an objective function for each available element; said selecting an available element which maximizes a ratio of increase; said adding the selected element to the summary; said updating gains of elements not in the summary; and said updating a residual budget.

8. In a digital medium environment including one or more computing devices to perform document summarization, a method comprising:
receiving a multimedia document to be processed to generate a summary;
ascertaining whether a budget is available for processing the multimedia document, the budget placing a constraint on the size of the summary and serving as a constraint on an amount of text content and image content that the summary is permitted to contain;
ascertaining whether a particular element of text content or image content in the multimedia document exists within the budget available such that the constraint on the amount of text content and image content that the summary is permitted to contain has not been met;
responsive to the particular element existing within the budget available, computing a value of an objective function for each available element in the multimedia document within the budget, the objective function providing a measure of a contribution of the particular element to coverage of information content relative to content of the multimedia document, diversity of information content relative to content of the multimedia document, and cohesion of information content relative to content of the summary, and being configured to compute a value for at least one text element and at least one image element in the multimedia document;
responsive to said computing, selecting an element in the multimedia document whose computed value maximizes a ratio of increase in the objective function to a cost associated with the selected element in the multimedia document wherein the cost of the element is a measure of the size of the element; and
adding the selected element to the summary.

9. The method as described in claim 8 , further comprising:
after adding the selected element to the summary,
updating a gain for each element in the multimedia document;
updating the budget;
ascertaining whether any budget is available for processing the multimedia document;

responsive to an element existing within the budget available iterating through said computing the value, said selecting an element, said adding the selected element, said updating a gain, said updating the budget and said ascertaining whether any budget is available until no element in the multimedia document is left unused having at least a cost lower than the available budget.

10. The method as described in claim 9 , wherein the objective function includes a term that measures coverage of image parts of the document.

11. The method as described in claim 9 , wherein the objective function includes a term that measures coverage of text parts of the document and a term that measures coverage of image parts of the document.

12. The method as described in claim 9 , wherein the objective function includes a term that provides a diversity reward which measures the amount of diverse information provided in the summary for both text parts and image parts.

13. The method as described in claim 9 , wherein the objective function includes:
a term that measures coverage of text parts of the document;
a term that measures coverage of image parts of the document; and
a term that provides a diversity reward which measures the amount of diverse information provided in the summary for both text parts and image parts.

14. In a digital medium environment having one or more computing devices to perform document summarization, a system comprising:
a processor;
a computer-readable storage media storing computer-readable instructions which, when executed by the processor, perform operations comprising:
receiving a multimedia document to be processed to generate a summary;
ascertaining whether a budget is available for processing the multimedia document, the budget placing a constraint on the size of the summary and serving as a constraint on an amount of text content and image content that the summary is permitted to contain;
processing the multimedia document by:
ascertaining whether a particular element of text content or image content in the multimedia document exists within the budget available such that the constraint on the amount of text

Assignees

• Adobe Inc

Inventors

• Modani Natwar
• Subramanian Vaishnavi
• Utpal
• Gupta Shivani
• Maneriker Pranav R
• Hiranandani Gaurush
• Sinha Atanu R

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title