Identifying intent in visual analytical conversations

Patent metadata

Field	Value
Publication number	US-12536382-B2
Application number	US-202318381162-A
Country	US
Kind code	B2
Filing date	Oct 17, 2023
Priority date	Dec 13, 2017
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

A method uses natural language processing for visual analysis of a dataset by a computer. The computer displays a data visualization based on a dataset retrieved from a database. The computer computes an initial visualization state that includes elements corresponding to data attributes of the data visualization. The computer receives user input to specify a natural language command related to the displayed data visualization. The computer extracts cue phrases from the natural language command. The computer also determines a transitional goal, to transform the data visualization, based on the cue phrases. The computer derives an updated visualization state based on the transitional goal, by applying transitional rules to each element of the initial visualization state. The computer subsequently computes analytical functions associated with the visualization states, thereby creating functional phrases. The computer then updates the data visualization based on the functional phrases.

First claim

What is claimed is:

1 . A method of using natural language for visual analysis of a dataset, comprising:
at a computer device having one or more processors and memory storing one or more programs configured for execution by the one or more processors:
displaying an initial data visualization based on a first dataset retrieved from a database using a first set of one or more queries;
determining an initial visualization state that includes a plurality of elements corresponding to (i) data attributes of the initial data visualization and (ii) visual encodings that map the data attributes to visual attributes in the initial data visualization;
receiving a user input to specify a natural language command related to the initial data visualization;
determining a transitional goal, to transform the initial data visualization, based on one or more cue phrases extracted from the natural language command;
deriving an updated visualization state based on the transitional goal by applying one or more transitional rules to the plurality of elements of the initial visualization state, wherein the one or more transitional rules prioritize a request directed to a first element, of the plurality of elements, that is explicitly specified in the natural language command over an inference that is derived from the one or more cue phrases, wherein:
the request directed to the first element includes a visual encoding request for a first data attribute; and
the method includes in accordance with a determination that the visual encoding request results in an empty set for the updated visualization state, disregarding the visual encoding request and applying one or more default heuristic rules;

computing a set of one or more analytical functions associated with the updated visualization state, thereby creating a set of one or more functional phrases;
updating the initial data visualization based on the set of one or more functional phrases to generate an updated data visualization; and
displaying the updated data visualization.

2 . The method of claim 1 , wherein:
the plurality of elements includes transformations to generate derived attributes from the data attributes; and deriving the updated visualization state includes performing a transformation operation that includes at least one of:
binning a quantitative variable;
changing a default ordering of the data attributes on an x-axis and/or a y-axis of the initial data visualization;
adding a redundant color encoding; and
generating a custom encoding.

3 . The method of claim 1 , wherein the transitional goal is selected from the group consisting of: elaboration, retrying, adjustment, undoing, or starting anew.

4 . The method of claim 1 , wherein the one or more transitional rules maintain conversational coherence between the initial visualization state and the updated visualization state by maintaining coherence in the data attributes and the visual encodings between the initial data visualization and the updated data visualization.

5 . The method of claim 1 , further comprising:
determining whether the one or more cue phrases contain a name attribute, a value, a term of measure, and/or a term of aggregation; and in accordance with a determination that the one or more cue phrases contain a name attribute or a value, but neither a term of measure nor a term of aggregation, determining the transitional goal to be calculating a count of records corresponding to the name attribute or the value.

6 . The method of claim 1 , further comprising:
determining whether the data attributes include a categorical attribute having a number of categories less than a threshold value; and in accordance with the determination that the data attributes include categorical attribute having a number of distinct categories less than the threshold value, determining the transitional goal to be comparing the categories.

7 . The method of claim 1 , further comprising:
determining if the one or more cue phrases contains a term corresponding to a chart type or a named visual variable; in accordance with a determination that the one or more cue phrases contains a term corresponding to a chart type, determining the transitional goal to be selecting the chart type; and in accordance with a determination that the one or more cue phrases contains a term corresponding to a named visual variable, determining the transitional goal to be encoding for the named visual variable according to the one or more cue phrases.

8 . The method of claim 1 , wherein the one or more transitional rules comprise a CONTINUE rule that maintains all the elements of the initial visualization state in the updated visualization state and adding one or more new elements in the updated visualization state based on the one or more cue phrases.

9 . The method of claim 1 , wherein the one or more transitional rules comprise a RETAIN rule that retains all the elements of the initial visualization state in the updated visualization state without adding any element based on the one or more cue phrases.

10 . The method of claim 1 , wherein the one or more transitional rules comprise a SHIFT rule that maintains all the elements of the initial visualization state in the updated visualization state and replacing one or more elements in the updated visualization state based on the one or more cue phrases.

11 . The method of claim 1 , wherein the one or more transitional rules comprise a RESET rule that clears all the elements of the initial visualization state to create an empty set.

12 . The method of claim 3 , further comprising:
determining if the one or more cue phrases contains terms that signify elaboration; and in accordance with a determination that the one or more cue phrases contains terms that signify elaboration, determining the transitional goal to elaborate.

13 . The method of claim 3 , further comprising:
determining if the one or more cue phrases contains terms that signify adjust/pivot; and in accordance with a determination that the one or more cue phrases contains terms that signify adjust/pivot, determining the transitional goal to adjust/pivot.

14 . The method of claim 3 , further comprising:
determining if the one or more cue phrases contains terms that signify undoing, or a repetition of an utterance that generated a desired visualization state; and in accordance with a determination that the one or more cue phrases contains terms that signify undoing, determining the transitional goal to undo.

15 . The method of claim 3 , further comprising:
determining if the one or more cue phrases contains terms that signify starting anew, or corresponds to an explicit reset; and in accordance with a determination that the one or more cue phrases contains terms that signify starting anew, determining the transitional goal to start anew.

16 . The method of claim 3 , further comprising:
determining if the one or more cue phrases contains terms that signify retry; and in accordance with a determination that the one or more cue phrases contains terms that signify retry, determining the transitional goal to retry.

17 . The method of claim 1 , wherein (i) elaboration implies adding new information in the updated data visualization, (ii) adjustment implies adapting one or more aspects of the initial data visualization in the updated data visualization, (iii) retrying implies re-attempting a previous step that failed, and (iv) undoing implies reverting the initial data visualization to a previous state.

18 . An elec

Assignees

• Tableau Software Llc

Inventors

• Tory Melanie K
• Setlur Vidya R

Classifications

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/289 Primary

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title