Multi-task machine learning for predicted touch interpretations

Patent metadata

Field	Value
Publication number	US-10261685-B2
Application number	US-201615393611-A
Country	US
Kind code	B2
Filing date	Dec 29, 2016
Priority date	Dec 29, 2016
Publication date	Apr 16, 2019
Grant date	Apr 16, 2019

Abstract

The present disclosure provides systems and methods that leverage machine learning to predict multiple touch interpretations. In particular, the systems and methods of the present disclosure can include and use a machine-learned touch interpretation prediction model that has been trained to receive touch sensor data indicative of one or more locations of one or more user input objects relative to a touch sensor at one or more times and, in response to receipt of the touch sensor data, provide one or more predicted touch interpretation outputs. Each predicted touch interpretation output corresponds to a different type of predicted touch interpretation based at least in part on the touch sensor data. Predicted touch interpretations can include a set of touch point interpretations, a gesture interpretation, and/or a touch prediction vector for one or more future times.

First claim

What is claimed is:

1. A computing device that determines touch interpretation from user input objects, comprising:
at least one processor;
a machine-learned touch interpretation prediction model, wherein the touch interpretation prediction model has been trained to directly receive raw touch sensor data indicative of measured touch sensor readings across a grid of points generated in response to one or more locations of one or more user input objects relative to a touch sensor at one or more times and, in response to receipt of the raw touch sensor data, output simultaneous multiple predicted touch interpretations, each predicted touch interpretation corresponding to a different type of predicted touch interpretation based at least in part on the touch sensor data, wherein the simultaneous multiple predicted touch interpretations comprise at least first and second predicted touch interpretations determined from a group comprising a set of touch point interpretations that respectively describe one or more intended touch points, a gesture interpretation that characterizes the set of touch point interpretations as a gesture determined from a predefined gesture class, and a touch prediction vector that describes one or more predicted future locations of the one or more user input objects respectively for one or more future times; and
at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
obtaining a first set of raw touch sensor data indicative of one or more user input object locations relative to a touch sensor over time;
inputting the first set of raw touch sensor data into the machine-learned touch interpretation prediction model;
receiving, as an output of the touch interpretation prediction model, simultaneous multiple predicted touch interpretations that describe predicted intentions of the one or more user input objects, each predicted touch interpretation describing a different predicted aspect of the one or more user input objects; and
performing one or more actions associated with the multiple predicted touch interpretations.

2. The computing device of claim 1 , wherein the one or more predicted touch interpretations comprise at least a first predicted touch interpretation comprising a set of touch point interpretations that respectively describe one or more intended touch points and a second predicted touch interpretation comprising a gesture interpretation that characterizes the set of touch point interpretations as a gesture determined from a predefined gesture class.

3. The computing device of claim 1 , wherein the machine-learned touch interpretation prediction model comprises a deep recurrent neural network with a plurality of output layers, each output layer corresponding to a different type of touch interpretation describing one or more predicted intentions of the one or more user input objects.

4. The computing device of claim 1 , wherein obtaining the first set of raw touch sensor data comprises obtaining the first set of raw touch sensor data associated with one or more fingers or hand portions of a user or a stylus operated by the user, the first set of raw touch sensor data descriptive of a location of the one or more fingers, hand portions or stylus relative to a touch-sensitive screen.

5. The computing device of claim 1 , wherein obtaining the first set of raw touch sensor data comprises:
obtaining the first set of raw time sensor data that provides at least one value describing a change in the location of the one or more user input objects in an x dimension, at least one value describing a change in the location of the one or more user input objects in a y dimension, and at least one value describing a change in time; or
obtaining the first set of raw time sensor data that provides at least two values describing at least two locations of the one or more user input objects in the x dimension, at least two values describing at least two locations of the one or more user input objects in the y dimension, and at least two values describing at least two times.

6. The computing device of claim 1 , wherein the machine-learned touch interpretation prediction model has been trained based on a first set of training data that includes a first portion of data corresponding to recorded touch sensor data indicative of one or more user input object locations relative to a touch sensor and a second portion of data corresponding to labels of determined touch interpretations applied to recorded screen content, wherein the first portion of data and the screen content are recorded at the same time.

7. The computing device of claim 1 , wherein the machine-learned touch interpretation prediction model has been trained based on a second set of training data that includes a first portion of data corresponding to an initial sequence of touch sensor data observations and a second portion of data corresponding to a subsequent sequence of touch sensor data observations.

8. The computing device of claim 1 , wherein the one or more predicted touch interpretations comprise a set of touch point interpretations that respectively describe zero, one or more touch points.

9. The computing device of claim 8 , wherein the set of touch point interpretations further comprises one or more of a touch type describing a predicted type of user input object associated with each touch point and an estimated pressure of the touch at each touch point.

10. The computing device of claim 1 , wherein the one or more predicted touch interpretations comprise a gesture interpretation that characterizes at least a portion of the first set of raw touch sensor data as a gesture determined from a predefined gesture class.

11. The computing device of claim 1 , wherein the one or more predicted touch interpretations comprise a touch prediction vector that describes one or more predicted future locations of the one or more user input objects respectively for one or more future times.

12. The computing device of claim 1 , wherein:
the machine-learned touch interpretation prediction model comprises at least one shared layer and multiple different and distinct output layers positioned structurally after the at least one shared layer; and
receiving, as an output of the touch interpretation prediction model, one or more predicted touch interpretations that describe predicted intentions of the one or more user input objects comprises receiving the one or more predicted touch interpretations from the multiple different and distinct output layers of the machine-learned touch interpretation prediction model.

13. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining data descriptive of a machine-learned touch interpretation prediction model, wherein the touch interpretation prediction model has been trained to receive touch sensor data indicative of one or more locations of one or more user input objects relative to a touch sensor at one or more times and, in response to receipt of the touch sensor data, provide simultaneous multiple predicted touch interpretation, each predicted touch interpretation corresponding to a different type of predicted touch interpretation based at least in part on the touch sensor data;
obtaining a first set of touch sensor data indicative of one or more user input object locations relative to a touch sensor over time;
inputting the

Assignees

• Google Inc
• Google Llc

Inventors

• Deselaers Thomas
• Carbune Victor

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F3/045

  using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title