Note recognition for overlapping physical notes

Patent metadata

Field	Value
Publication number	US-10296789-B2
Application number	US-201615213598-A
Country	US
Kind code	B2
Filing date	Jul 19, 2016
Priority date	Oct 16, 2013
Publication date	May 21, 2019
Grant date	May 21, 2019

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A computing device includes a processor, an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes, and an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image. The image processing engine determines a boundary of each note in the plurality of overlapping physical notes in the input image, and generates a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

First claim

The invention claimed is:

1. A method comprising:
receiving, by a computing device, an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors;
processing the input image with the computing device to identify each of the overlapping physical notes in the input image, wherein identifying the overlapping physical notes comprises:
classifying pixels of the input image into one of a plurality of color classes, wherein each of the different background colors corresponds to a respective color class of the plurality of color classes, wherein classifying the pixels comprises:
grouping adjacent pixels having the same color class into regions that each correspond to a note of the plurality of overlapping physical notes in the input image; and
determining, based on boundaries of the regions and according to a machine-learning pattern recognition algorithm, boundaries of each of the overlapping physical notes in the input image;

extracting, by the computing device, content for each of the respective overlapping physical notes based on the respective determined boundary;
creating, by the computing device, a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and
associating, by the computing device, the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

2. The method of claim 1 , further comprising:
applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and
generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

3. The method of claim 1 , further comprising:
determining, by the computing device, whether each of the plurality of overlapping physical notes in the input image comprises a border color different from the background color;
upon determining that the plurality of overlapping physical notes includes one or more physical notes with a border color, applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and
generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

4. The method of claim 2 , further comprising:
displaying, by the computing device, the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and
displaying, by the computing device, the plurality of segmented digital notes on a second portion of the graphical user interface.

5. The method of claim 1 , further comprising:
determining, by the computing device, whether at least one of the overlapping physical notes in the input image includes one or more fiducial markers using a marker detection module;
based on the determination that at least one of the overlapping physical notes in the input image includes one or more fiducial markers, detecting one or more fiducial markers which indicate the size and shape of the note;
determining, by the computing device and based on the fiducial markers, the boundary of each of the physical notes in the plurality of overlapping physical notes;
segmenting, by the computing device and based on the determined boundaries, each of the physical notes in the plurality of overlapping physical notes;
generating, by the computing device, a plurality of segmented digital notes corresponding to the plurality of segmented physical notes.

6. The method of claim 1 , wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the size and shape of each note in the plurality of overlapping physical notes.

7. The method of claim 1 , wherein the machine-learning pattern recognition algorithm includes a support vector machine.

8. A non-transitory computer-readable medium comprising instructions for causing a programmable processor to:
receive an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors;
process the input image to identify each of the overlapping physical notes in the input image, wherein the instructions that cause the processor to identify the overlapping physical notes comprise instructions that cause the processor to:
classify pixels of the input image into one of a plurality of color classes, wherein each of the different background colors corresponds to a respective color class of the plurality of color classes, wherein the instructions that cause the processor to classify the pixels comprise instructions that cause the processor to:
group adjacent pixels having the same color class into regions that each correspond to a note of the plurality of overlapping physical notes in the input image; and
determine, based on boundaries of the regions and according to a machine-learning pattern recognition algorithm, boundaries of each of the overlapping physical notes in the input image;

extract content for each of the respective overlapping physical notes based on the respective determined boundary;
generate a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and
associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

9. The non-transitory computer-readable medium of claim 8 , wherein the machine-learning pattern recognition algorithm includes a support vector machine.

10. A computing device comprising:
a processor; and
a memory storing instructions that, when executed, cause the processor to:
receive an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors;
process the input image with the computing device to identify the plurality of overlapping physical notes in the input image, wherein, to identify the plurality of overlapping physical notes, the processor is configured to:
classify pixels of the input image into one of a plurality of color classes, wherein each of the different background colors corresponds to a respective color class of the plurality of color classes, wherein, to classify the pixels, the processor is configured to;
group adjacent pixels having the same color class into regions that each correspond to a note of the plurality of overlapping physical notes in the input image; and
determine, based on boundaries of the regions and according to a machine-learning pattern recognition algorithm, boundaries of each of the overlapping physical notes in the input image;

extract content for each of the respective overlapping physical notes based on the respective determined boundary;
generate a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and
associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of

Assignees

• 3M Innovative Properties Co

Inventors

• Somasundaram Guruprasad
• Sivalingam Ravishankar
• Mahli David M
• Moran Cristin E
• Moore Richard J
• Axelsson Pontus
• Rotstein Michael
• Persson Roland

Classifications

• H04L51/08 Primary

  Annexed information, e.g. attachments · CPC title

• G06V30/414

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06F40/171

  by use of digital ink · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06V10/245

  by locating a pattern; Special marks for positioning · CPC title