Enhancing documents portrayed in digital images

Patent metadata

Field	Value
Publication number	US-11017159-B2
Application number	US-202016830047-A
Country	US
Kind code	B2
Filing date	Mar 25, 2020
Priority date	Jul 22, 2016
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

The present disclosure is directed toward systems and methods that efficiently and effectively generate an enhanced document image of a displayed document in an image frame captured from a live image feed. For example, systems and methods described herein apply a document enhancement process to a displayed document in an image frame that result in an enhanced document image that is cropped, rectified, un-shadowed, and with dark text against a mostly white background. Additionally, systems and method described herein determine whether a stored digital content item includes a displayed document. In response to determining that a stored digital content item does include a displayed document, systems and methods described herein generate an enhanced document image of a displayed document included in the stored digital content item.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
a non-transitory computer-readable medium storing instructions thereon that, when executed by the at least one processor, cause the system to:
provide a graphical user interface comprising a set of selectable options, the set of selectable options comprising a first option for scanning a document;
provide a live camera image feed in response to receiving an indication that a user selected the first option;
in response to detecting a displayed document within the live camera image feed, providing, for display within the live camera image feed, a live document boundary indicator associated with the displayed document;
capture, from the live camera image feed, an image frame that comprises the displayed document to generate a modified image frame that excludes one or more portions of the image frame displayed outside of the live document boundary indicator; and
process the modified image frame to generate an enhanced document image corresponding to the displayed document, the enhanced document image comprising one or more visual alterations to the displayed document.

2. The system as recited in claim 1 , wherein the set of selectable options further comprises a second option for capturing a photo.

3. The system as recited in claim 1 , wherein generating the modified image frame comprises:
detecting the one or more portions of the image frame that are outside of the live document boundary indicator; and
cropping the image frame to remove the one or more portions of the image frame that are outside of the live document boundary indicator.

4. The system as recited in claim 1 , wherein processing the modified image frame to generate the enhanced document image comprises at least one of: rectifying the displayed document, converting the displayed document to grayscale, or denoising the displayed document.

5. The system as recited in claim 1 , wherein processing the modified image frame to generate the enhanced document image comprises correcting visual flaws within the modified image frame, wherein correcting the visual flaws comprises at least one of: removing shadows, correcting foreground content, or generating a uniform background within the displayed document.

6. The system as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate an additional modified image frame based on capturing, from the live camera image feed, an additional image frame comprising an additional displayed document;
process the additional modified image frame to generate an additional enhanced document image corresponding to the additional displayed document; and
combine the additional enhanced document image and the enhanced document image to create a single file in a document file format.

7. The system as recited in claim 6 , wherein the document file format comprises a portable document format.

8. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computer device to:
provide a graphical user interface comprising a set of selectable options, the set of selectable options comprising an option for scanning a document;
provide, for presentation within the graphical user interface, a live camera image feed in response to receiving an indication that a user selected the option for scanning a document;
in response to detecting a displayed document within the live camera image feed, providing, for display within the live camera image feed, a live document boundary indicator associated with the displayed document;
capture, from the live camera image feed, an image frame that comprises the displayed document to generate a modified image frame that excludes one or more portions of the image frame displayed outside of the live document boundary indicator; and
process the modified image frame to generate an enhanced document image corresponding to the displayed document, the enhanced document image comprising one or more visual alterations to the displayed document.

9. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to upload the enhanced document image in a document file format to an online content management system in an account associated with the computing device.

10. The non-transitory computer-readable medium recited in claim 8 , wherein generating the modified image frame comprises:
detecting the one or more portions of the image frame that are outside of the live document boundary indicator; and
cropping the image frame to remove the one or more portions of the image frame that are outside of the live document boundary indicator.

11. The non-transitory computer-readable medium recited in claim 8 , wherein processing the modified image frame to generate the enhanced document image comprises correcting visual flaws within the modified image frame, wherein correcting the visual flaws comprises at least one of: removing shadows, correcting foreground content, or generating a uniform background within the displayed document.

12. The non-transitory computer-readable medium recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computer device to:
generate an additional modified image frame based on capturing, from the live camera image feed, an additional image frame comprising an additional displayed document;
process the additional modified image frame to generate an additional enhanced document image corresponding to the additional displayed document; and
combine the additional enhanced document image and the enhanced document image to create a single file in a document file format.

13. The non-transitory computer-readable medium recited in claim 12 , wherein the document file format comprises a portable document format.

14. The non-transitory computer-readable medium recited in claim 12 , further comprising instructions that, when executed by the at least one processor, cause the computer device to share the enhanced document image in the document file format with a client device associated with a user by sending the enhanced document image to the client device.

15. A method comprising:
providing a graphical user interface comprising a set of selectable options, the set of selectable options comprising a first option for scanning a document;
provide a live camera image feed in response to receiving an indication that a user selected the first option;
in response to detecting a displayed document within the live camera image feed, providing, for display within the live camera image feed, a live document boundary indicator associated with the displayed document;
capture, from the live camera image feed, an image frame that comprises the displayed document to generate a modified image frame that excludes one or more portions of the image frame displayed outside of the live document boundary indicator; and
process the modified image frame to generate an enhanced document image corresponding to the displayed document, the enhanced document image comprising one or more visual alterations to the displayed document.

16. The method as recited in claim 15 , wherein the set of selectable options further comprises a second option for capturing a photo.

17. The method as recited in claim 15 , wherein generating the modified image frame comprises:
detecting the one or more portions of the ima

Assignees

• Dropbox Inc

Inventors

• Welinder Nils Peter
• Belhumeur Peter N
• Xiong Ying
• Baek Jongmin
• Kozlov Simon
• Berg Thomas
• Kriegman David J

Classifications

• G06F40/123 Primary

  Storage facilities · CPC title

• G06F40/166 Primary

  Editing, e.g. inserting or deleting · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06V10/242

  by image rotation, e.g. by 90 degrees · CPC title

• G06V10/30

  Noise filtering · CPC title