Eye gaze for spoken language understanding in multi-modal conversational interactions

Patent metadata

Field	Value
Publication number	US-10901500-B2
Application number	US-201916399414-A
Country	US
Kind code	B2
Filing date	Apr 30, 2019
Priority date	Sep 25, 2014
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

Improving accuracy in understanding and/or resolving references to visual elements in a visual context associated with a computerized conversational system is described. Techniques described herein leverage gaze input with gestures and/or speech input to improve spoken language understanding in computerized conversational systems. Leveraging gaze input and speech input improves spoken language understanding in conversational systems by improving the accuracy by which the system can resolve references—or interpret a user's intent—with respect to visual elements in a visual context. In at least one example, the techniques herein describe tracking gaze to generate gaze input, recognizing speech input, and extracting gaze features and lexical features from the user input. Based at least in part on the gaze features and lexical features, user utterances directed to visual elements in a visual context can be resolved.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a user utterance of a user derived from speech input referring to a visual element of multiple visual elements in a visual context for displaying the multiple visual elements;
receiving, from a tracking device, a gaze input with respect to the user;
identifying, for each visual element of the multiple visual elements in the visual context, ter associated with the visual element;
computing, for each visual element of the multiple visual elements independent of the gaze input, a lexical similarity between the text associated with the visual element and the user utterance and, for each lexical similarity, a lexical probability corresponding to the referred visual element;
extracting a gaze feature using the gaze input and the multiple visual elements;
determining that a particular visual element of the multiple visual elements is the referred visual element in the user utterance, using a lexical probability of the computed lexical probabilities together with a computed probability that the particular visual element is the referred visual element in the speech input using the gaze feature; and
causing an action associated with the particular visual element to be performed.

2. The computer-implemented method of claim 1 , wherein using the lexical probability of the computed lexical probabilities together with the computed probability using the gaze feature includes generating a probability that the particular visual element is the referred visual element in the speech input given the gaze feature and given one or more of a cosine similarity between term vectors of the text associated with the particular visual element and the speech input and a number of characters in a longest common subsequence of the particular visual element and the speech input.

3. The computer-implemented method of claim 1 , comprising:
filtering each visual element of the multiple visual elements using the computed lexical probabilities together with the computed probability using the gaze feature;
identifying a set of visual elements of the multiple visual elements having probabilities above a predetermined threshold; and
identifying the particular visual element from the set of visual elements.

4. The computer-implemented method of claim 3 , wherein filtering each visual element of the multiple visual elements includes ranking the visual elements using, for each visual element, a probability that the respective visual element is the referred visual element given the gaze feature and the computed lexical similarity for the respective visual element.

5. The computer-implemented method of claim 1 , wherein the method includes multiplying the lexical probability and the computed probability using the gaze feature together to form a new probability that the particular visual element is the referred visual element in the speech input.

6. The computer-implemented method of claim 1 , comprising:
identifying a plurality of fixation points associated with the gaze input;
grouping a predetermined number of the plurality of fixation points together in a cluster; and
identifying a centroid of the cluster as a specific fixation point for extracting the gaze feature from the gaze input.

7. The computer-implemented method of claim 6 , comprising:
computing a start time and an end time of the speech input; and
extracting the gaze feature based at least in part on:
distances between the specific fixation point and an area associated with individual visual elements of the multiple visual elements;
the start time of the speech input; and
the end time of the speech input.

8. A device comprising:
one or more processors;
computer-readable media encoded with instructions that, when executed by the one or more processors, cause the device to perform operations comprising:
receiving a user utterance of a user derived from speech input referring to a visual element of multiple visual elements in a visual context for displaying the multiple visual elements;
receiving, from a tracking device, a gaze input with respect to the user;
identifying, for each visual element of the multiple visual elements in the visual context, text associated with the visual element;
computing, for each visual element of the multiple visual elements independent of the gaze input; a lexical similarity between the text associated with the visual element and the user utterance and, for each lexical similarity, a lexical probability corresponding to the referred visual element;
extracting a gaze feature using the gaze input and the multiple visual elements;
determining that a particular visual element of the multiple visual elements is the referred visual element in the user utterance, using a lexical probability of the computed lexical probabilities together with a computed probability that the particular visual element is the referred visual element in the speech input using the gaze feature; and
causing an action associated with the particular visual element to be performed.

9. The device of claim 8 , wherein the operations comprise:
determining a bounding box for each visual element of the multiple visual elements, the bounding box comprising an area associated with the respective visual element; and
computing, in the extracting of the gaze feature, distances between bounding boxes for the visual elements and fixation points associated with the gaze input at predetermined times.

10. The device of claim 8 , wherein the gaze feature comprises one or more of how frequently the user looked at a bounding box during the speech input with the bounding box comprising an area associated with an individual visual element; a total length of time the user looked at the bounding box during the speech input; how frequently the bounding box was within a predetermined radius of a centroid fixation point during the speech input; and a total length of time the bounding box was within a predetermined radius of a centroid fixation point during the speech input.

11. The device of claim 8 , wherein the operations include multiplying the lexical probability and the computed probability using the gaze feature together to form a new probability that the particular visual element is the referred visual element in the speech input.

12. The device of claim 8 , wherein the visual context is a free-form web browser or an application interface.

13. The device of claim 8 , wherein the action is caused to be performed in the visual context.

14. A device comprising:
one or more processors;
computer-readable media encoded with instructions that, when executed by the one or more processors, cause the device to perform operations comprising:
identifying visual elements in a visual context for displaying the visual elements;
receiving a user utterance of a user derived from speech input referring to a visual element of the visual elements in the visual context;
receiving a gaze input with respect to the user and the visual elements;
extracting a lexical feature based at least in part on computing lexical similarities between the user utterance and text associated with individual visual elements of the visual elements in the visual context;
extracting a gaze feature using the gaze input and the visual elements;
determining that a particular visual element of the visual elements is the referred visual element in the user utterance, using at least in part a computation based on the gaze feature and the lexical feature; and
causing an action associated with the particular visual element to be performed, wherein determining that the part

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Prokofieva Anna
• Celikyilmaz Fethiye Asli
• Hakkani-Tur Dilek Z
• Heck Larry
• Slaney Malcolm

Classifications

• G10L17/22

  Interactive procedures; Man-machine interfaces · CPC title

• G10L15/08

  Speech classification or search · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title