Note capture and recognition with manual assist

Patent metadata

Field	Value
Publication number	US-9292186-B2
Application number	US-201514608955-A
Country	US
Kind code	B2
Filing date	Jan 29, 2015
Priority date	Jan 31, 2014
Publication date	Mar 22, 2016
Grant date	Mar 22, 2016

Abstract

At least some aspects of the present disclosure feature a computing device configured to receive an input image of an environment having a plurality of physical notes. The computing device automatically processes the input image to identify at least some of the plurality of the physical notes in the input image and displays the input image and indications indicative of the identified physical notes on a user interface. The computing device receives a user input indicating a position within the input image via a user interface and, responsive to the user input, recognizes proximate to the position a missed one of the physical notes that was not identified by the computing device when initially processing the input image.

First claim

What is claimed is:

1. A computing device having a processor, the computing device comprising:
an image collection module configured to receive an input image of a plurality of physical notes;
a note identification module, implemented by the processor, configured to process the input image to identify at least some of the plurality of physical notes in the input image; and
a user interface configured to display the input image and indications indicative of the identified physical notes,
wherein the user interface is further configured to receive a user input indicating a position in the input image, and
wherein the note identification module is further configured to identify, based on the position, a missed one of the physical notes in the input image that was not identified by the note identification module when processing the input image.

2. The computing device of claim 1 , wherein the note identification module is further configured to apply, responsive to the user input, a color segmentation algorithm to determine a boundary of the missed physical note based on the position in the input image.

3. The computing device of claim 1 , wherein the note identification module is further configured to apply, responsive to the user input, a marching algorithm to determine a boundary of the missed physical note based on the position in the input image.

4. The computing device of claim 1 , wherein the note identification module is further configured to determine an area of interest based on areas of the identified notes in the input image, and wherein the note identification module is further configured to determine, responsive to the user input, a boundary of the missed physical note based on the position in the input image and the area of interest.

5. The computing device of claim 1 , wherein the note identification module is further configured to identify a plurality of candidate missed physical notes in the input image based on the position, and generate a score for each candidate missed physical note based on at least one factor selected from a group consisting of area, aspect ratio, position, and color values, and
wherein the note identification module is further configured to choose a candidate missed physical note having a highest score among the plurality of candidate missed physical notes as the missed physical note.

6. The computing device of claim 1 , wherein the note identification module is further configured to determine a color of the missed physical note and identify one or more other missed physical notes based on the color of the missed physical note.

7. The computing device claim 1 , wherein the note identification module is further configured to generate digital notes, each digital note corresponding to a note of the plurality of notes identified in the input image.

8. The computing device of claim 1 , wherein the user interface is further configured to display the input image overlaid with an indication indicative of the missed physical note.

9. The computing device of claim 1 , wherein the input image comprises at least one of a static image, an animated image, a sequence of images, and a video.

10. The computing device of claim 1 , wherein the input image depicts a visual representation of a scene having the plurality of physical notes.

11. A method of capturing and recognizing notes implemented on a computing device having a processor, a memory, and a user interface, the method comprising:
receiving an input image of a plurality of physical notes;
identifying, by the processor, at least some of the plurality of physical notes in the input image;
displaying the input image and indications indicative of the identified physical notes on the user interface;
receiving a user input indicating a position in the input image; and
identifying, by the processor and responsive to the position indicated by the user input, a missed physical note in the input image that was not identified by the processor when initially processing the input image.

12. The method of claim 11 , further comprising:
applying, by the processor, a color segmentation algorithm to determine a boundary of the missed physical note based on the position in the input image.

13. The method of claim 11 , further comprising:
applying, by the processor, a marching algorithm to determine a boundary of the missed physical note based on the position in the input image.

14. The method of claim 11 , further comprising:
determining an area of interest based on areas of the identified notes in the input image; and
determining a boundary of the missed physical note based on the position in the input image and the area of interest.

15. The method of claim 11 , further comprising:
identifying a plurality of candidate missed physical notes in the image based on the position;
generating a score for each candidate missed physical note based on at least one factor selected from a group consisting of area, aspect ratio, position, and color values; and
choosing a candidate missed physical note having a highest score among the plurality of candidate missed physical notes as the missed physical note.

16. The method of claim 11 , further comprising:
determining a color of the missed physical note; and
identifying one or more other missed physical notes based on the color of the missed physical note.

17. The method of claim 11 , further comprising:
generating digital notes, each digital note representing a note identified in the input image.

18. The method of claim 11 , further comprising:
displaying the input image overlaid with an indication indicative of the missed physical note on the user interface.

19. The method claim 11 , wherein the input image is overlaid with indications indicative of the identified notes.

20. The method of claim 11 , wherein the input image depicts a visual representation of a scene having the plurality of physical notes.

Assignees

• 3M Innovative Properties Co

Inventors

• Moore Richard J
• Mahli David M
• Axelsson Pontus
• Moran Cristin E
• Persson Roland
• Rotstein Michael
• Sivalingam Ravishankar
• Somasundaram Guruprasad
• Wolk Diane R
• Åkerlund Linus

Classifications

• G06V30/414 Primary

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06F3/04845 Primary

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F3/03

  Arrangements for converting the position or the displacement of a member into a coded form · CPC title

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title