Personalized summary generation of data visualizations

Patent metadata

Field	Value
Publication number	US-10909313-B2
Application number	US-201715630462-A
Country	US
Kind code	B2
Filing date	Jun 22, 2017
Priority date	Jun 22, 2016
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

Various embodiments are generally directed to systems for summarizing data visualizations (i.e., images of data visualizations), such as a graph image, for instance. Some embodiments are particularly directed to a personalized graph summarizer that analyzes a data visualization, or image, to detect pre-defined patterns within the data visualization, and produces a textual summary of the data visualization based on the pre-defined patterns detected within the data visualization. In various embodiments, the personalized graph summarizer may include features to adapt to the preferences of a user for generating an automated, personalized computer-generated narrative. For instance, additional pre-defined patterns may be created for detection and/or the textual summary may be tailored based on user preferences. In some such instances, one or more of the user preferences may be automatically determined by the personalized graph summarizer without requiring the user to explicitly indicate them. Embodiments may integrate machine learning and computer vision concepts.

First claim

The invention claimed is:

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:
identify a data visualization comprising a graph image;
determine a set of graph-type correlation scores for the graph image, the set of graph-type correlation scores to include a graph-type correlation score for each graph type of a plurality of graph types, each graph-type correlation score based on a comparison of at least a portion of the graph image with one or more graph-type models associated with each graph type of the plurality of graph types;
evaluate the set of graph-type correlation scores to identify a graph type of the graph image;
retrieve a set of patterns based on the graph type of the graph image, each pattern in the set of patterns to include one or more pattern examples;
determine a set of region of interest correlation scores for the graph image based on matching the one or more pattern examples of each pattern in the set of patterns with at least a portion of the graph image, the set of region of interest correlation scores to include at least one region of interest correlation score for each pattern in the set of patterns;
evaluate the set of region of interest correlation scores to identify one or more candidate regions of interest of the graph image, each of the one or more candidate regions of interest to include a portion of the graph image;
retrieve a set of pattern models based on the set of candidate regions of interest of the graph image, each candidate region of interest in the set of candidate regions of interest associated with one pattern model in the set of pattern models, and each pattern model in the set of pattern models associated with one pattern in the set of patterns;
compare each candidate region of interest in the set of candidate regions of interest to an associated pattern model in the set of pattern models to determine a set of pattern model correlation scores, the set of pattern model correlation scores to include a pattern model correlation score for each candidate region of interest of the one or more candidate regions of interest;
identify one or more detected patterns based on the set of pattern model correlation scores;
retrieve one or more text templates from a computer-readable storage medium based on the one or more detected patterns, the one or more text templates to include at least one portion of text associated with each detected pattern of the one or more detected patterns, each text template of the one or more text templates associated with a priority level;
arrange the one or more text templates in an order based on the priority level associated with each text template to generate a textual description of the graph image; and
generate a personalized summary of the graph image based on the textual description with the one or more text templates ordered based on the priority level associated with each text template.

2. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
detect a portion of the graph image with contextual information;
extract a textual element from the portion of the graph image with contextual information; and
insert at least a portion of the textual element extracted from the portion of the graph image with contextual information into at least one text template of the one or more text templates to generate the textual description of the graph image.

3. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
identify a component of the graph image based on the graph type;
detect a portion of the graph image with potential contextual information; and
determine contextual information is absent from the portion of the graph image with potential contextual information based on the component of the graph image identified based on the graph type.

4. The apparatus of claim 1 , matching a pattern example of a pattern in the set of patterns with at least a portion of the graph image comprising:
overlay at least a portion of the pattern example on the graph image in a plurality of positions; and
compute a region of interest correlation score in the set of region of interest correlation scores for each of the plurality of positions.

5. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
receive an additional pattern example; and
update a pattern model in the set of pattern models based on the additional pattern example.

6. The apparatus of claim 1 , each pattern model correlation score to indicate a likelihood of a respective candidate region of interest of the one or more candidate regions of interest including an associated pattern.

7. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
present the one or more text templates arranged based on the priority level associated with each template sentence via a user interface;
arrange the one or more text templates in an updated order based on input received via the user interface;
alter a priority level of at least one of the one or more text templates based on the updated order; and
generate the textual description of the graph image based on the priority level associated with each text template, the priority level associated with each text template to include the priority level of the at least one of the one or more text templates altered based on the updated order.

8. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
alter the priority level of a text template based on the input received via a user interface.

9. The apparatus of claim 1 , at least one pattern in the set of patterns comprising a personalized pattern, wherein the processor is caused to perform operations comprising:
create the personalized pattern based on one or more example graph images and one or more pattern examples identified in the example graph images based on input received via a user interface.

10. The apparatus of claim 9 , wherein the processor is caused to perform operations comprising:
associate one or more of a priority level, a template sentence, or a graph type with the personalized pattern based on input received via the user interface.

11. A computer-implemented method, comprising:
identifying a data visualization comprising a graph image;
determining a set of graph-type correlation scores for the graph image, the set of graph-type correlation scores to include a graph-type correlation score for each graph type of a plurality of graph types, each graph-type correlation score based on a comparison of at least a portion of the graph image with one or more graph-type models associated with each graph type of the plurality of graph types;
evaluating the set of graph-type correlation scores to identify a graph type of the graph image;
retrieving a set of patterns based on the graph type of the graph image, each pattern in the set of patterns to include one or more pattern examples;
determining a set of region of interest correlation scores for the graph image based on matching the one or more pattern examples of each pattern in the set of patterns with at least a portion of the graph image, the set of region of interest correlation scores to include at least one region of interest correlation score for each pattern in the set of patterns;
evaluating the set of region of interest correlation scores to identify one or more candidate regions of interest of the graph image, each of the one or more candidate regions of interest to include a portion of the graph image;
retr

Assignees

• Sas Inst Inc

Inventors

• Can Ethem F
• Crowell Richard W
• Tetterton James
• Peterson Jared
• Sethi Saratendu

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06F40/186 Primary

  Templates · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title