Detecting ink gestures based on spatial and image data processing

Patent metadata

Field	Value
Publication number	US-11587346-B2
Application number	US-202017117151-A
Country	US
Kind code	B2
Filing date	Dec 10, 2020
Priority date	Dec 10, 2020
Publication date	Feb 21, 2023
Grant date	Feb 21, 2023

Abstract

Ink-processing technology is set forth herein for detecting a gesture that a user performs in the course of interacting with an ink document. The technology operates by identifying a grouping of ink strokes created by the user. The technology then determines whether the grouping expresses a gesture based on a combination of spatial information and image information, both of which describe the grouping. That is, the spatial information describes a sequence of positions traversed by the user in drawing the grouping of ink strokes using an ink capture device, while the image information refers to image content in an image produced by rendering the grouping into image form. The technology also provides a technique for identifying the grouping by successively expanding a region of analysis, to ultimately provide a spatial cluster of ink strokes for analysis.

First claim

What is claimed is:

1. A computing system for detecting a gesture that a user performs in a course of interacting with an ink document, comprising:
hardware logic circuitry including one or more hardware processors and/or one or more configurable gate units, configured to perform the operations of:
receiving a first ink stroke most recently drawn by the user, and adding the first ink stroke to a first grouping of ink strokes that initially includes just the first ink stroke;
identifying a first spatial scope of analysis that is large enough to encompass the first ink stroke;
extending the first spatial scope of analysis;
identifying any other ink stroke that is included in the first spatial scope of analysis as extended, and adding said any other ink stroke to the first grouping of ink strokes;
generating a first classification result based on the first grouping of ink strokes;
receiving a second ink stroke drawn by the user, and adding the second ink stroke to a second grouping of ink strokes that initially includes just the second ink stroke;
identifying a second spatial scope of analysis that is large enough to encompass the second ink stroke;
extending the second spatial scope of analysis;
identifying any other ink stroke that is included in the second spatial scope of analysis as extended, and adding said any other ink stroke identified by extending the second spatial scope as extended to the second grouping of ink strokes;
generating a second classification result based on the second grouping of ink strokes, independent of the first classification result; and
performing an operation corresponding to a particular gesture for a case in which a corresponding classification result indicates that a corresponding group of ink strokes contains the particular gesture.

2. The computing system of claim 1 , wherein the first grouping of ink strokes is a first spatial cluster of ink strokes that omits at least one ink stroke that the user produced between a first-captured ink stroke in the first spatial cluster and a last-captured ink stroke in the first spatial cluster.

3. The computing system of claim 1 , wherein the method further comprises repeating said extending the first spatial scope of analysis and identifying any other ink stroke in the first spatial scope of analysis as extended until no additional ink stroke is identified or until another termination condition is reached.

4. The computing system of claim 3 , wherein one termination condition is reached when a size of the first spatial scope of analysis achieves a prescribed threshold value.

5. The computing system of claim 1 ,
wherein the first spatial scope of analysis after entry of the first ink stroke is a bounding box that encompasses the first ink stroke, and
wherein said extending of the first spatial scope of analysis is performed by expanding at least one dimension of the bounding box.

6. The computing system of claim 5 , wherein said identifying any other ink stroke in the first spatial scope of analysis as extended involves determining whether said any other ink stroke in the first spatial scope of analysis intersects the bounding box.

7. The computing system of claim 1 , wherein said identifying any other ink stroke in the first spatial scope of analysis as extended involves determining whether said any other ink stroke in the first spatial scope of analysis as extended intersects an already-existing ink stroke in the first grouping of ink strokes.

8. The computing system of claim 1 , wherein said extending of the first spatial scope of analysis expands the first spatial scope of analysis by a fixed amount, independent of context.

9. The computing system of claim 1 , wherein the particular gesture is a document-editing gesture, and the operation performed for the particular gesture is a document-editing action.

10. The computing system of claim 1 , wherein the operations further include:
identifying spatial information that describes a sequence of positions traversed by the user in drawing the first grouping of ink strokes using the ink capture device;
converting the first grouping of ink strokes into image information;
mapping the spatial information into a first feature embedding; and
mapping the image information into a second feature embedding,
wherein the generating the first classification result is based on a combination of the first feature embedding and the second feature embedding,
wherein said mapping the spatial information into a first feature embedding uses a first neural network provided by the hardware logic circuitry, and
wherein said mapping the image information into a second feature embedding uses a second neural network provided by the hardware logic circuitry.

11. The computing system of claim 1 , wherein the operations further include:
identifying spatial information that describes a sequence of positions traversed by the user in drawing the first grouping of ink strokes using the ink capture device;
converting the first grouping of ink strokes into image information;
mapping the spatial information into a first feature embedding; and
mapping the image information into a second feature embedding,
wherein the generating the first classification result is based on a combination of the first feature embedding and the second feature embedding,
wherein the generating the first classification result further comprises:
combining the first feature embedding and the second feature embedding to produce a combined feature embedding;
mapping the combined embedding to an output feature embedding using a neural network provided by the hardware logic circuitry; and
identifying the first classification result based on the output feature embedding.

12. The computing system of claim 11 , wherein said combining is performed by concatenating the first feature embedding and the second feature embedding.

13. A method for detecting a gesture that a user performs in a course of interacting with an ink document, comprising:
receiving a first ink stroke most recently drawn by the user using an ink capture device, within the ink document;
identifying a first spatial scope of analysis that is large enough to encompass the first ink stroke;
extending the first spatial scope of analysis;
identifying any other ink stroke that is included in the first spatial scope of analysis as extended, and adding said any other ink stroke to the first grouping of ink strokes generating a first classification result based on the first grouping of ink strokes;
receiving a second ink stroke drawn by the user, and adding the second ink stroke to a second grouping of ink strokes that initially includes just the second ink stroke;
identifying a second spatial scope of analysis that is large enough to encompass the second ink stroke;
extending the second spatial scope of analysis;
identifying any other ink stroke that is included in the second spatial scope of analysis as extended, and adding said any other ink stroke in the second spatial scope of analysis to the second grouping of ink strokes;
generating a second classification result based on the second grouping of ink strokes, independent of the first classification result; and
performing an operation based on a particular classification result for a case in which the particular classification result indicates that a corresponding grouping of ink strokes contains the particular gesture,
the method defining a stateless algorithm in which the second grouping of ink strokes is processed independently of processing of the first grouping of ink strokes.

14. The method of claim 13 ,
wherein the

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Solomon Oz
• Elachqar Oussama
• Doroshenko Sergey Aleksandrovich
• Mohajerin Nima
• Yadav Badal

Classifications

• G06V30/32 Primary

  Digital ink · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V30/347

  Sampling; Contour coding; Stroke extraction · CPC title

• G06F18/253

  of extracted features · CPC title

• G06V30/36 Primary

  Matching; Classification · CPC title