Note recognition for overlapping physical notes

Patent metadata

Field	Value
Publication number	US-2016328857-A1
Application number	US-201615213598-A
Country	US
Kind code	A1
Filing date	Jul 19, 2016
Priority date	Oct 16, 2013
Publication date	Nov 10, 2016
Grant date	—

Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A computing device includes a processor, an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes, and an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image. The image processing engine determines a boundary of each note in the plurality of overlapping physical notes in the input image, and generates a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

First claim

1 . A method comprising:
receiving, by a computing device, an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors; processing the input image with the computing device to identify each of the overlapping physical notes in the input image including determining a boundary of each of the overlapping physical notes in the input image; extracting, by the computing device, content for each of the respective overlapping physical notes based on the respective determined boundary; creating, by the computing device, a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and associating, by the computing device, the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

2 . The method of claim 1 , further comprising:
applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

3 . The method of claim 1 , further comprising:
determining, by the computing device, whether each of the plurality of overlapping physical notes in the input image comprises a border color different from the background color; upon determining that the plurality of overlapping physical notes includes one or more physical notes with a border color, applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

4 . The method of claim 2 , further comprising:
displaying, by the computing device, the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and displaying, by the computing device, the plurality of segmented digital notes on a second portion of the graphical user interface.

5 . The method of claim 1 , further comprising:
determining, by the computing device, whether at least one of the overlapping physical notes in the input image includes one or more fiducial markers using a marker detection module; based on the determination that at least one of the overlapping physical notes in the input image includes one or more fiducial markers, detecting one or more fiducial markers which indicate the size and shape of the note; determining, by the computing device and based on the fiducial markers, the boundary of each of the physical notes in the plurality of overlapping physical notes; segmenting, by the computing device and based on the determined boundaries, each of the physical notes in the plurality of overlapping physical notes; generating, by the computing device, a plurality of segmented digital notes corresponding to the plurality of segmented physical notes.

6 . The method of claim 1 , wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the size and shape of each note in the plurality of overlapping physical notes.

7 . The method of claim 1 , wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the plurality of physical notes according to color spaces associated with background colors of the plurality of notes.

8 . The method of claim 1 , wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the plurality of notes according to a pattern recognition algorithm.

9 . The method of claim 8 , wherein the pattern recognition algorithm includes at least one of a group consisting of:
a support vector machine; a cross-correlation; and a template matching.

10 . A non-transitory computer-readable medium comprising instructions for causing a programmable processor to:
receive an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors; process the input image to identify each of the overlapping physical notes in the input image including determining a boundary of each of the overlapping physical notes in the input image; extract content for each of the respective overlapping physical notes based on the respective determined boundary; generate a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes; and associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

11 . A computing device comprising:
a processor; an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes, each of the overlapping physical notes having a background color, adjacent ones of the plurality of physical notes having different background colors; and an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image, determine a boundary of each note in the plurality of overlapping physical notes in the input image, extract content for each of the respective overlapping physical notes based on the respective determined boundary, and generate a plurality of digital notes, each of the digital notes representing a particular one of the identified overlapping physical notes, and associate the extracted content for each respective physical note of the identified overlapping physical notes with a corresponding digital note of the plurality of digital notes that represents the respective physical note.

12 . The device of claim 11 , wherein the image collection module includes an image capture device configured to capture the input image.

13 . The device of claim 11 , wherein the image collection module includes a communication interface configured to receive the input image.

14 . The device of claim 11 , wherein the image processing engine is further configured to:
apply a color segmenting process to segment each physical note in the plurality of overlapping physical notes; and generate a plurality of segmented digital notes corresponding to the plurality of segmented notes.

15 . The device of claim 14 , where the image processing engine further configured to:
display the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and display the plurality of segmented digital notes on a second portion of the graphical user interface.

16 . The device of claim 11 , wherein the image processing engine is further configured to:
determine whether the plurality of overlapping physical notes in the input image comprise a different border color; upon determining that the plurality of overlapping physical notes includes two or more physi

Assignees

• 3M Innovative Properties Co

Inventors

• Somasundaram Guruprasad
• Sivalingam Ravishankar
• Mahli David M
• Moran Cristin E
• Moore Richard J
• Axelsson Pontus
• Rotstein Michael
• Persson Roland

Classifications

• H04L51/08 Primary

  Annexed information, e.g. attachments · CPC title

• G06V30/414

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06F40/171

  by use of digital ink · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06V10/245

  by locating a pattern; Special marks for positioning · CPC title