Live document detection in a captured video stream

Patent metadata

Field	Value
Publication number	US-11017158-B2
Application number	US-201916457423-A
Country	US
Kind code	B2
Filing date	Jun 28, 2019
Priority date	Jul 22, 2016
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

The present disclosure is directed toward systems and methods to quickly and accurately identify boundaries of a displayed document in a live camera image feed, and provide a document boundary indicator within the live camera image feed. For example, systems and methods described herein utilize different display document detection processes in parallel to generate and provide a document boundary indicator that accurately corresponds with a displayed document within a live camera image feed. Thus, a user of the mobile computing device can easily see whether the document identification system has correctly identified the displayed document within the camera viewfinder feed.

First claim

What is claimed is:

1. A method comprising:
providing a graphical user interface comprising a live camera feed;
detecting a visual representation of a physical document within the live camera feed;
providing, within the graphical user interface, a live document boundary indicator associated with the visual representation of the physical document, the live document boundary indicator visually comprising a first boundary shape corresponding to the visual representation of the physical document at a first document position within the live camera feed;
detecting, for the visual representation of the physical document, a change from the first document position to a second document position within the live camera feed; and
modifying, within the live camera feed of the graphical user interface, the live document boundary indicator to visually comprise a second boundary shape corresponding to the visual representation of the physical document at the second document position.

2. The method as recited in claim 1 , further comprising analyzing a first image frame from the live camera feed with a quadrilateral detection process to detect the first document position.

3. The method as recited in claim 2 , further comprising determining the first boundary shape corresponding to the visual representation of the physical document at the first document position based on the quadrilateral detection process.

4. The method of claim 3 , further comprising generating a first boundary indicator position and a first boundary indicator shape for the live document boundary indicator.

5. The method as recited in claim 4 , further comprising overlaying, on the first image frame of the live camera feed, the live document boundary indicator having the first boundary indicator position and the first boundary indicator shape.

6. The method as recited in claim 2 , further comprising analyzing a second image frame of the live camera feed based on a second process that is different than the quadrilateral detection process.

7. The method as recited in claim 1 , wherein detecting the change from the first document position to the second document position within the live camera feed comprises:
receiving one or more movement signals associated with a mobile computing device comprising a camera that is generating the live camera feed; and
predicting the second document position based on the one or more movement signals.

8. The method as recited in claim 1 , wherein detecting the change from the first document position to the second document position within the live camera feed comprises:
receiving one or more movement signals associated with a mobile computing device comprising a camera that is generating the live camera feed; and
predicting the second boundary shape based on the one or more movement signals.

9. The method as recited in claim 1 , further comprising:
generating, for the live document boundary indicator, a first boundary indicator position corresponding to the first document position and a first boundary indicator shape corresponding to the first boundary shape; and
overlaying, on the live camera feed, the live document boundary indicator having the first boundary indicator position and the first boundary indicator shape.

10. A computing device comprising:
at least one processor; and
at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the computing device to:
provide a graphical user interface comprising a live camera feed;
detect a visual representation of a physical document within the live camera feed;
provide, within the graphical user interface, a live document boundary indicator associated with the visual representation of a physical document, the live document boundary indicator visually comprising a first boundary shape corresponding to the visual representation of the physical document at a first document position within the live camera feed;
detect a change from the first document position to a second document position within the live camera feed; and
modify, within the live camera feed of the graphical user interface, the live document boundary indicator to visually comprise a second boundary shape corresponding to the visual representation of the physical document at the second document position.

11. The computing device as recited in claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computing device to analyze a first image frame from the live camera feed with a quadrilateral detection process to detect the first document position.

12. The computing device as recited in claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
analyze a second image frame of the live camera feed; and
predict the second document position based on analyzing the second image frame.

13. The computing device as recited in claim 11 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
analyze a second image frame of the live camera feed; and
predict the second boundary shape based on analyzing the second image frame.

14. The computing device as recited in claim 10 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate, for the live document boundary indicator, a first boundary indicator position corresponding to the first document position and a first boundary indicator shape corresponding to the first boundary shape.

15. The computing device as recited in claim 14 , further comprising instructions that, when executed by the at least one processor, cause the computing device to overlay, on the live camera feed, the live document boundary indicator having the first boundary indicator position and the first boundary indicator shape.

16. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
provide a graphical user interface comprising a live camera feed;
detect a visual representation of a physical document within the live camera feed;
provide, within the graphical user interface, a live document boundary indicator associated with the visual representation of a physical document, the live document boundary indicator visually comprising a first boundary shape corresponding to the visual representation of the physical document at a first document position within the live camera feed;
detect a change from the first document position to a second document position within the live camera feed; and
modify, within the live camera feed of the graphical user interface, the live document boundary indicator to visually comprise a second boundary shape corresponding to the visual representation of the physical document at the second document position.

17. The non-transitory computer-readable medium as recited in claim 16 , further comprising instructions that, when executed by the at least one processor, cause the computing device to analyze a first image frame from the live camera feed with a quadrilateral detection process to detect the first document position.

18. The non-transitory computer-readable medium as recited in claim 17 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
analyze a second image frame of the live camera feed; and
predict the second document position

Assignees

• Dropbox Inc

Inventors

• Welinder Nils Peter
• Belhumeur Peter N
• Xiong Ying
• Baek Jongmin
• Kozlov Simon
• Berg Thomas
• Kriegman David J

Classifications

• G06F40/123 Primary

  Storage facilities · CPC title

• G06F40/166 Primary

  Editing, e.g. inserting or deleting · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06V10/242

  by image rotation, e.g. by 90 degrees · CPC title

• G06V10/30

  Noise filtering · CPC title