Enhancing documents portrayed in digital images

Patent metadata

Field	Value
Publication number	US-11620439-B2
Application number	US-202117323838-A
Country	US
Kind code	B2
Filing date	May 18, 2021
Priority date	Jul 22, 2016
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

The present disclosure is directed toward systems and methods that efficiently and effectively generate an enhanced document image of a displayed document in an image frame captured from a live image feed. For example, systems and methods described herein apply a document enhancement process to a displayed document in an image frame that result in an enhanced document image that is cropped, rectified, un-shadowed, and with dark text against a mostly white background. Additionally, systems and method described herein determine whether a stored digital content item includes a displayed document. In response to determining that a stored digital content item does include a displayed document, systems and methods described herein generate an enhanced document image of a displayed document included in the stored digital content item.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
a non-transitory computer-readable medium storing instructions thereon that, when executed by the at least one processor, cause the system to:
provide a live camera image feed within a graphical user interface, the live camera image feed comprising a partially displayed document that has one or more document portions outside of an image frame within the live camera feed;
capture, from the live camera image feed, the image frame comprising the partially displayed document;
estimate the location of the one or more document portions outside of the image frame based on displayed document portions within the image frame to generate an enhanced document image comprising an estimated display of a full document corresponding to the partially displayed document; and
convert the enhanced document image into a document file.

2. The system as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to estimate the location of the one or more document portions outside of the image frame based on the displayed document portions within the image frame by projecting a document corner location outside the image frame using one or more of a document edge within the image frame or a document corner within the image frame.

3. The system as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to estimate the location of the one or more document portions outside of the image frame based on the displayed document portions within the image frame by projecting a document edge location outside the image frame using one or more of a document edge within the image frame or a document corner within the image frame.

4. The system as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to estimate the location of the one or more document portions outside of the image frame based on the displayed document portions within the image frame by:
projecting a height corresponding to the partially displayed document;
projecting a width corresponding to the partially displayed document; and
identifying coordinates of one or more document corner locations outside the image frame based on the projected height and projected width.

5. The system as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to provide the enhanced document image for display within the graphical user interface prior to converting the enhanced document image into a document file format.

6. The system as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to:
prior to converting the enhanced document image to a document file format:
provide, via the graphical user interface, user editing options corresponding to the enhanced document image; and
modify the enhanced document image based on receiving an indication of a user edit.

7. The system as recited in claim 1 , wherein the document file format comprises a portable document format.

8. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computer device to:
capture, from a live camera image feed, an image frame comprising a partially displayed document, the partially displayed document comprising displayed document portions within the image frame and one or more document portions outside of the image frame;
estimate the location of the one or more document portions outside of the image frame based on the displayed document portions within the image frame to generate an enhanced document image comprising an estimated display of a full document corresponding to the partially displayed document; and
convert the enhanced document image into a document file format.

9. The non-transitory computer-readable medium recited in claim 8 , wherein generating the enhanced document image further comprises applying a uniform background color to the enhanced document image.

10. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to provide the enhanced document image for display within the graphical user interface prior to converting the enhanced document image into a document file.

11. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to modify the enhanced document image based on receiving an indication of a user edit prior to converting the enhanced document image into a document file.

12. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to estimate the location of the one or more document portions outside of the image frame by projecting the coordinates of one or more document corner locations outside the image frame.

13. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to estimate the location of the one or more document portions outside of the image frame by projecting the locations of one or more document edge locations outside the image frame.

14. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to upload the document file to an online content management system.

15. A method comprising:
providing a live camera image feed within a graphical user interface, the live camera image feed comprising a partially displayed document that has one or more document portions outside of an image frame within the live camera feed;
capturing, from the live camera image feed, the image frame comprising the partially displayed document; and
estimating the location of the one or more document portions outside of the image frame based on displayed document portions within the image frame to generate an enhanced document image comprising an estimated display of a full document corresponding to the partially displayed document.

16. The method as recited in claim 15 , further comprising providing a preview of the enhanced document image for display within the graphical user interface.

17. The method as recited in claim 16 , further comprising providing user editing options within the graphical user interface that allow a user to edit the preview of the enhanced document image.

18. The method as recited in claim 17 , further comprising:
modifying the enhanced document image based on receiving an indication of a user edit; and
converting the modified enhanced document image into a document file.

19. The method as recited in claim 15 , further comprising uploading the enhanced document image in a document file format to an online content management system.

20. The method as recited in claim 15 , wherein generating the enhanced document image further comprises at least one of: rectifying the estimated display of the full document, converting the estimated display of the full document to grayscale, denoising the estimated display of the full document, or adjusting a color

Assignees

• Dropbox Inc

Inventors

• Welinder Nils Peter
• Belhumeur Peter N
• Xiong Ying
• Baek Jongmin
• Kozlov Simon
• Berg Thomas
• Kriegman David J

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F40/123 Primary

  Storage facilities · CPC title