Simulated transparent display with augmented reality for remote collaboration

Patent metadata

Field	Value
Publication number	US-2016353055-A1
Application number	US-201615167011-A
Country	US
Kind code	A1
Filing date	May 27, 2016
Priority date	May 29, 2015
Publication date	Dec 1, 2016
Grant date	—

Abstract

A method and system for remote collaboration and remote instruction utilizing computing devices at trainee site and mentor sites are disclosed. Annotations are superimposed onto a trainee's view of a view field displayed using a simulated transparent display with augmented reality (“AR”). By tracking the position of the trainee's view point, the position of the trainee's computing device, and the geometry of the view field, the annotations remain anchored to the image of the view field when the trainee device moves and/or when the view field deforms or becomes occluded.

First claim

What is claimed is:

1 . A system comprising:
a first computing device, the first display device comprising:
a first processor;
a first camera coupled to or operable by the first processor, the first camera configured to capture a reference image and a current image of a view field;
a display configured to display the current image;
a depth acquisition module coupled to or operable by the first processor, the depth acquisition module configured to acquire geometry information of the view field; and
a first transceiver coupled to or operable by the first processor, the first transceiver configured to:
transmit the reference image and the geometry information; and
receive annotation information to be displayed on the first display; and

a second computing device communicatively connectable with the first computing device, the second computing device comprising:
a second processor;
a user-operable input device coupled or operable by to the second processor;
a second transceiver coupled or operable by to the second processor, the second transceiver configured to receive the reference image and the geometry information from the first computing device;
an annotation module coupled to or operable by the second processor to receive input via the input device and determine at least one annotation based at least in part on the input; and
a second display coupled to or operable by the second processor, the second display configured to display the reference image and to display the at least one annotation superimposed over the reference image;
wherein:
the second transceiver is further configured to transmit reference frame data to the first computing device, the reference frame data associated, at least in part, with the reference image, the geometry information, and the at least one annotation; and
the first computing device is further configured to display the at least one annotation anchored to the displayed current image based on the received reference frame data.

2 . The system of claim 1 , wherein the annotation module is further configured to display a plurality of selectable symbols on the second display, each of the plurality of selectable symbols associated with a predefined annotation.

3 . The system of claim 1 , wherein the second computing device further comprises:
a reference feature detector module coupled to or operable by the second processor, the reference feature detector configured to detect a plurality of image features in the reference image relative to the at least one annotation based, at least in part, on the geometry information; and a reference descriptor extractor module coupled to or operable by the second processor, the reference descriptor extractor module configured to determine a descriptor for each of the plurality of the image features, wherein the reference frame data comprises at least some of the descriptors of the reference image.

4 . The system of claim 3 , wherein the first computing device further comprises:
a current feature detector module coupled to or operable by the first processor, the current feature detector configured to detect a plurality of image features in the current image of the view field; a current descriptor extractor module coupled to or operable by the first processor, the current descriptor extractor module configured to compute a descriptor for each of the plurality of the image features in the current image; a descriptor matching module coupled to or operable by the first processor, the descriptor matching module configured to match at least some of the current image descriptors with at least some of the reference image descriptors to provide match information; a mapping module coupled to or operable by the first processor, the mapping module configured to:
determine a mapping for the annotation based on the match information; and
transform the annotation from the reference image to the current image based on the determined mapping; and
an annotation rendering module coupled to or operable by the first processor, the annotation rendering module configured to render the transformed annotation over the current image displayed on the first display.

5 . The system of claim 4 , wherein the first computing device further comprises:
a tracking module coupled to or operable by the first processor, the tracking module configured to:
determine a position of a view point of a user of the first computing device; and
determine an updated image of the view field based at least in part on the current image of the view field, the geometry information, and the position of the view point of the user;
wherein the first computing device is configured to display on the first display the updated image of the view field.

6 . The system of claim 5 , wherein the tracking module comprises at least one of:
at least one camera facing the user, or a depth measurement device facing the user.

7 . The system of claim 5 , wherein the tracking module is further configured to determine a position of the first computing device relative to the view point of a user and to determine the updated image further based on the position of the first computing device and the geometry information.

8 . A method comprising:
acquiring a reference image and geometry information of operating view field by a first computing device; transmitting the reference image from the first computing device to a second computing device; displaying the reference image at the second computing device; determining an annotation at the second computing device; displaying the annotation over the reference image on a display of at the second computing device; generating reference frame data at the second computing device based, at least in part, on the reference image, the geometry information, and the annotation, the reference frame data comprising information associated with a plurality of image features in the reference image; transmitting the reference frame data and the annotation from the second computing device to the first computing device; receiving the reference frame data and the annotation by the first computing device; acquiring a current image of the view field by the first computing device; determining a position of the annotation in the current image based on the reference frame data; displaying the current image on a display of the first computing device; and displaying the annotation over the current image at the determined position on the display of the first computing device.

9 . The method of claim 8 , wherein the determining the annotation comprises at least one of:
receiving path information via a user-operable input device of the second computing device and determining the annotation based at least in part on the path information; displaying a plurality of symbols on the display of the second computing device and receiving, via the input device, a selection of a symbol of the plurality of symbols and determining the annotation as a predefined annotation associated with the symbol; or receiving textual input via the input device and determining the annotation based at least in part on the textual input.

10 . The method of claim 8 , wherein the generating the reference frame data comprises:
detecting at least some of the plurality of image features in the reference image relative to the annotation based, at least in part, on the geometry information; and determining the information including a descriptor for each of the at least some of the plurality of the image features.

11 . The method of claim 10 , wherein determining the position of the annotation comprises:
detecting a plurality of image

Assignees

• Purdue Research Foundation

Inventors

• Popescu Voicu S
• Wachs Juan P

Classifications

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• A61B2090/368

  changing the image on a display according to the operator's position · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• A61B90/37 Primary

  Surgical systems with images on a monitor during operation · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title