Parsing an ink document using object-level and stroke-level processing

Patent metadata

Field	Value
Publication number	US-11514695-B2
Application number	US-202017117152-A
Country	US
Kind code	B2
Filing date	Dec 10, 2020
Priority date	Dec 10, 2020
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

Technology is described herein for parsing an ink document having a plurality of ink strokes. The technology performs stroke-level processing on the plurality of ink strokes to produce stroke-level information, the stroke-level information identifying at least one characteristic associated with each ink stroke. The technology also performs object-level processing on individual objects within the ink document to produce object-level information, the object-level information identifying one or more groupings of ink strokes in the ink document. The technology then parses the ink document into constituent parts based on the stroke-level information and the object-level information. In some implementations, the technology converts the ink stroke data into an ink image. The stroke-level processing and/or the object-level processing may operate on the ink image using one or more neural networks. More specifically the stroke-level processing can classify pixels in the input image, while the object-level processing can identify bounding boxes containing possible objects.

First claim

What is claimed is:

1. A computing system for parsing ink stroke data, comprising:
hardware logic circuitry including one or more hardware processors and/or one or more configurable gate units, configured to perform the operations of:
receiving the ink stroke data from an ink capture device, the ink stroke data describing a plurality of ink strokes in an ink document;
converting the ink stroke data into an ink image that is made up of a plurality of pixels;
performing stroke-level processing on the plurality of ink strokes in the ink document to produce stroke-level information, the stroke-level information identifying a classification for each individual pixel in the ink image, each particular classification of a particular pixel indicating a most likely type of object to which the particular pixel corresponds;
performing object-level processing on individual objects within the ink document to produce object-level information, the object-level information identifying a plurality of regions in the ink document that are likely to include a respective plurality of objects, each object of the plurality of objects including a grouping of ink strokes; and
parsing the ink document into constituent parts based on the stroke-level information and the object-level information, the parsing including:
determining a stroke-based classification associated with a particular ink stroke based on the stroke-level information, the stroke-based classification identifying a most likely type of object to which the particular ink stroke corresponds;
determining, based on the object-level information, an object-based classification associated with a particular object in a particular region of the ink document identified by the object-level processing that includes a particular grouping of ink strokes, the object-based classification identifying a most likely type of the particular object in the particular region; and
determining whether the particular ink stroke is a member of the particular object based on a combination of the stroke-based classification and the object-based classification.

2. The computing system of claim 1 , wherein said performing stroke-level processing operates on the ink image, and wherein said performing object-level processing operates on the ink image.

3. The computing system of claim 1 ,
wherein the operations further include mapping the ink image into an image feature embedding, and
wherein said performing stroke-level processing and said performing object-level processing operate on the image feature embedding.

4. The computing system of claim 3 ,
wherein the hardware logic circuitry includes an encoder neural network that uses successive down-converting stages, and
wherein said mapping of the ink image into an image feature embedding uses the encoder neural network.

5. The computing system of claim 1 ,
wherein the hardware logic circuitry includes a pixel classifier neural network,
wherein said performing stroke-level processing classifies individual pixels in the ink image using the pixel classifier neural network, and
wherein said parsing includes, as a part thereof, classifying the particular ink stroke based on classifications of pixels that are included in the particular ink stroke.

6. The computing system of claim 5 , wherein the pixel classifier neural network performs said stroke-level processing to produce an output image in plural successive up-converting stages, the output image identifying classifications for respective pixels in the output image.

7. The computing system of claim 6 , wherein the object-level processing is performed for different image scales based on respective instances of image information generated by at least two of the successive up-converting stages of the pixel classifier neural network.

8. The computing system of claim 1 , wherein said parsing includes:
determining whether the particular ink stroke intersects the particular region associated with the particular object; and
determining whether the particular ink stroke is a member of the particular object based on a rule that specifies that: the particular ink stroke is a member of the particular object when the particular ink stroke intersects the particular region by at least a prescribed amount, and the particular ink stroke and the particular object have a same classification; and the particular ink stroke is not a member of the particular object when the particular ink stroke and the particular object do not have a same classification.

9. The computing system of claim 8 ,
wherein the particular ink stroke also intersects another region associated with another object,
wherein the other object has an object-based classification that differs from the stroke-based classification, and
wherein the rule assigns the particular ink stroke to the particular object that matches the stroke-based classification, and not the other object that does not match the stroke-based classification.

10. The computing system of claim 1 , wherein said determining the object-based classification determines the object-based classification by performing processing on the pixels in the ink image.

11. The computing system of claim 1 , wherein said determining the object-based classification determines the object-based classification based on stroke-based classifications of the ink strokes in the particular grouping.

12. The computing system of claim 1 ,
wherein the stroke-based classification or the object-based classification, but not both the stroke-based classification and the object-based classification, is established with at least a prescribed confidence level, and
and wherein said determining whether the particular ink stroke is a member of the particular object includes assigning the particular ink stroke to whatever classification, the stroke-based classification or the object-based classification, has been established with at least the prescribed confidence level.

13. The computing system of claim 1 ,
wherein the hardware logic circuitry includes an object-detector neural network, and
wherein said performing object-level processing uses the object-detector neural network to identify a plurality of bounding boxes in the ink image respectively associated with the plurality of regions.

14. A method for parsing ink stroke data, comprising:
receiving the ink stroke data from an ink capture device, the ink stroke data describing a plurality of ink strokes in an ink document;
converting the ink stroke data into an ink image that is made up of a plurality of pixels;
performing stroke-level processing on the plurality of ink strokes in the ink document to produce stroke-level information, the stroke-level information identifying classifications of individual pixel in the ink image;
performing object-level processing on individual objects within the ink document to produce object-level information, the object-level information identifying a plurality of regions in the ink document that are likely to include a respective plurality of objects, each object of the plurality of objects including a grouping of ink strokes, the object-level processing being different than the stroke-level processing; and
parsing the ink document into constituent parts based on the stroke-level information and the object-level information, the parsing including:
determining a stroke-based classification associated with a particular ink stroke based on the stroke-level information, the stroke-based classification identifying a most likely type of object to which the particular ink stroke corresponds;
determining, based on the object-level information, an o

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Elachqar Oussama
• Yadav Badal
• Solomon Oz
• Doroshenko Sergey Aleksandrovich
• Mohajerin Nima

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V30/18057 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V30/36

  Matching; Classification · CPC title

• G06V10/95

  structured as a network, e.g. client-server architectures · CPC title

• G06V30/347 Primary

  Sampling; Contour coding; Stroke extraction · CPC title