Syncing digital and physical documents

Patent metadata

Field	Value
Publication number	US-10706278-B2
Application number	US-201715650089-A
Country	US
Kind code	B2
Filing date	Jul 14, 2017
Priority date	Jul 14, 2017
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

Methods and systems for incorporating physical documents into a document review workflow involving electronic documents. One or more embodiments detect a presence of a physical document within a field of view of an AR device and map the physical document to an existing electronic document based on visual features of the physical document. Additionally, one or more embodiments determine at least one difference between the physical document and the electronic document and create, for the physical document and the electronic document a shared state mapping including the difference(s). One or more embodiments then apply the difference to the physical document or the electronic document by displaying the difference(s) in an AR layer within the field of view of the AR device or storing the difference(s) in the electronic document.

First claim

What is claimed is:

1. In a digital media environment for shared document synchronization, a computer-implemented method of leveraging augmented reality to integrate physical documents into a digital workflow, comprising:
detecting, by at least one processor of an augmented reality device, a presence of a physical document within a field of view of the augmented reality device;
analyzing, by the at least one processor, the physical document to identify at least one visual feature of the physical document;
mapping, by the at least one processor and based on the at least one visual feature, the physical document to an electronic document from a document repository;
determining, by the at least one processor using image processing, a plurality of differences between the physical document and the electronic document, wherein the physical document comprises a first difference of the plurality of differences and the electronic document comprises a second difference of the plurality of differences;
creating, by the at least one processor in response to determining the plurality of differences, a shared state mapping for the physical document and the electronic document, the shared state mapping being a temporary digital file indicating the plurality of differences between the physical document and the electronic document; and
applying, by the at least one processor, the shared state mapping to the physical document and the electronic document by writing the first difference to the electronic document and inserting the second difference into an augmented reality layer displayed over the physical document within the field of view of the augmented reality device.

2. The computer-implemented method as recited in claim 1 , further comprising:
analyzing a plurality of electronic documents in the document repository to identify a plurality of visual features for the plurality of electronic documents;
storing the plurality of visual features for the plurality of electronic documents in a feature database; and
identifying an electronic document that includes the at least one visual feature of the physical document by comparing the at least one visual feature of the physical document to the plurality of visual features in the feature database.

3. The computer-implemented method as recited in claim 2 , wherein identifying the electronic document that includes the at least one visual feature of the physical document further comprises determining, from the plurality of electronic documents, an electronic document that includes the at least one visual feature in a location of the electronic document corresponding to a location of the at least one visual feature within the physical document.

4. The computer-implemented method as recited in claim 2 , wherein analyzing the plurality of electronic documents in the document repository to identify the plurality of visual features for the plurality of electronic documents comprises:
converting each electronic document of the plurality of electronic documents to an image format; and
analyzing each electronic document of the plurality of electronic documents in the image format to extract image objects corresponding to the plurality of visual features.

5. The computer-implemented method as recited in claim 1 , wherein applying the shared state mapping to the physical document by inserting the second difference into the augmented reality layer displayed over the physical document within the field of view of the augmented reality device comprises:
creating a two-dimensional texture comprising the second difference; and
mapping the two-dimensional texture to the physical document in the augmented reality layer using coordinates of the physical document.

6. The computer-implemented method as recited in claim 1 , further comprising:
detecting a new annotation to the physical document in addition to the first difference;
updating the shared state mapping for the physical document and the electronic document with the new annotation; and
applying the updated shared state mapping to the electronic document by writing the new annotation to the electronic document.

7. The computer-implemented method as recited in claim 1 , further comprising:
detecting a new annotation to the electronic document in addition to the second difference;
updating the shared state mapping for the physical document and the electronic document with the new annotation; and
applying the updated shared state mapping to the physical document by inserting the new annotation into the augmented reality layer displayed over the physical document within the field of view of the augmented reality device.

8. The computer-implemented method as recited in claim 1 , further comprising:
providing an augmented reality tool for interacting with the augmented reality layer;
detecting a new annotation to the augmented reality layer in response to a user input using the augmented reality tool;
updating the shared state mapping for the physical document and the electronic document with the new annotation; and
applying the updated shared state mapping to the electronic document by writing the new annotation to the electronic document.

9. The computer-implemented method as recited in claim 1 , further comprising:
determining that the physical document leaves the field of view of the augmented reality device;
detecting that the physical document re-enters the field of view of the augmented reality device; and
generating a new shared state mapping for the physical document and the electronic document comprising any differences that were created between the physical document leaving the field of view and then reentering the field of view.

10. The computer-implemented method as recited in claim 1 , further comprising:
comparing the physical document to an unannotated version of the electronic document;
generating a first differences list comprising differences between the physical document and the unannotated version of the electronic document;
comparing an annotated version of the electronic document to the unannotated version of the electronic document;
generating a second differences list comprising differences between the annotated version of the electronic document and the unannotated version of the electronic document; and
creating the shared state mapping based on the first differences list and the second differences list.

11. The computer-implemented method as recited in claim 1 , wherein:
mapping the physical document to the electronic document from the document repository comprises determining that the physical document corresponds to a page from a plurality of pages of the electronic document; and
applying the shared state mapping to the electronic document comprises writing the first difference to the corresponding page of the electronic document.

12. In a digital media environment for shared document synchronization, a system for leveraging augmented reality to integrate physical documents into a digital workflow, the system comprising:
a non-transitory computer memory comprising a document repository comprising electronic documents; and
at least one computing device storing instructions thereon, that, when executed by the at least one computing device, cause the system to:
detect a presence of a physical document within a field of view of an augmented reality device;
map the physical document to an electronic document from the document repository based on a similarity of the physical document to the electronic document;
determine, using image processing, a plurality of differences between the physical document and the electronic document, wherein the phy

Assignees

• Adobe Inc

Inventors

• Marda Vikas
• Sheikh Roshni
• Sachan Kartik
• Singhal Aman

Classifications

• G06V30/418

  Document matching, e.g. of document images · CPC title

• G06Q10/10 Primary

  Office automation; Time management · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06T2219/004

  Annotating, labelling · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title