Remote expert system

Patent metadata

Field	Value
Publication number	US-9665985-B2
Application number	US-201414461252-A
Country	US
Kind code	B2
Filing date	Aug 15, 2014
Priority date	Aug 15, 2014
Publication date	May 30, 2017
Grant date	May 30, 2017

Abstract

A remote expert application identifies a manipulation of virtual objects displayed in a first wearable device. The virtual objects are rendered based a physical object viewed with a second wearable device. A manipulation of the virtual objects is received from the first wearable device. A visualization of the manipulation of the virtual objects is generated for a display of the second wearable device. The visualization of the manipulation of the virtual objects is communicated to the second wearable device.

First claim

What is claimed is:

1. A system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising:
receiving, from a first wearable device, data including a video feed captured by the first wearable device, the video feed depicting to a physical object viewed with the first wearable device;
comparing the data received from the first wearable device to a three dimensional (3D) model dataset of the physical object, yielding a comparison;
identifying, based on the comparison, an anomaly between the physical object viewed with the first wearable device and the 3D model dataset of the physical object;
identifying, based on the anomaly, a user designated as an expert related to the anomaly;
providing, to a second wearable device used by the user designated as the expert related to the anomaly, virtual objects displayed by the first wearable device, causing the second wearable device to present the virtual objects to the user designated as the expert;
receiving, from the second wearable device, data describing a manipulation of the virtual objects displayed by the second wearable device, the virtual objects displayed in the first wearable device in relation to a reference object viewed with the first wearable device;
generating a visualization of the manipulation of the virtual objects from the second wearable device; and
communicating the visualization of the manipulation of the virtual objects to the first wearable device, causing the first wearable device to present the visualization of the manipulation to a user of the first wearable device.

2. The system of claim 1 , the operations further comprising:
identifying the physical object from the video feed;
generating a 3D model of the virtual objects based on the identification of the physical object; and
communicating the 3D model of the virtual objects to the first wearable device, the first wearable device being configured to render the 3D model of the virtual objects in relation to the physical object.

3. The system of claim 1 , the operations further comprising recording the manipulation of the virtual objects from the second wearable device.

4. The system of claim 1 , wherein the manipulation of the virtual objects is determined by the second wearable device using a visual gesture module, the visual gesture module configured to determine a predefined area of a display of the second wearable device and a feature of the virtual objects, and to change a state of the feature in response to the feature being in the predefined area of the display of the second wearable device.

5. The system of claim 1 , wherein the manipulation of the virtual objects is determined by the second wearable device using a physical gesture module, the physical gesture module configured to determine a physical movement of a finger of the user of the second wearable device relative to the virtual objects displayed by the second wearable device.

6. The system of claim 1 , wherein the visualization includes an animation of the manipulation of the virtual objects.

7. The system of claim 1 , the operations further comprising:
receiving, from the first wearable device, a subsequent manipulation of the virtual objects; and
communicating a visualization of the subsequent manipulation of the virtual objects to the second wearable device, causing the second wearable device to present the subsequent manipulation to the user designated as the expert.

8. The system of claim 1 , the operations further comprising:
receiving video feeds from a plurality of wearable devices, the video feeds depicting the physical object;
generating a 3D model of the physical object based on the video feeds received from the plurality of wearable devices; and
updating the 3D model of physical object based on manipulations of the physical object identified from the video feeds.

9. A method comprising:
receiving, from a first wearable device, data including a video feed captured by the first wearable device, the video feed depicting a physical object viewed with the first wearable device;
comparing the data received from the first wearable device to a three dimensional (3D) model dataset of the physical object, yielding a comparison;
identifying, based on the comparison, an anomaly between the physical object viewed with the first wearable device and the 3D model dataset of the physical object;
identifying, based on the anomaly, a user designated as an expert related to the anomaly;
providing, to a second wearable device used by the user designated as the expert related to the anomaly, virtual objects displayed by the first wearable device, causing the second wearable device to present the virtual objects to the user designated as the expert;
receiving, from the second wearable device, data describing a manipulation of the virtual objects displayed by the second wearable device, the virtual objects displayed in the first wearable device in relation to a reference object viewed with the first wearable device;
generating a visualization of the manipulation of the virtual objects from the second wearable device; and
communicating the visualization of the manipulation of the virtual objects to the first wearable device, causing the first wearable device to present the visualization of the manipulation to a user of the first wearable device.

10. The method of claim 9 , further comprising:
identifying the physical object from the video feed;
generating a 3D model of the virtual objects based on the identification of the physical object; and
communicating the 3D model of the virtual objects to the first wearable device, the first wearable device being configured to render the 3D model of the virtual objects in relation to the physical object.

11. The method of claim 9 , further comprising:
recording the manipulation of the virtual objects from the second wearable device.

12. The method of claim 9 , wherein
the manipulation of the virtual objects is determined by the second wearable device using a visual gesture module, the visual gesture module configured to determine a predefined area of a display of the second wearable device and a feature of the virtual objects, and to change a state of the feature in response to the feature being in the predefined area of the display of the second wearable device.

13. The method of claim 9 , wherein
the manipulation of the virtual objects is determined by the second wearable device using a physical gesture module, the physical gesture module configured to determine a physical movement of a finger of the user of the second wearable device relative to the virtual objects displayed by the second wearable device.

14. The method of claim 9 , wherein the visualization includes an animation of the manipulation of the virtual objects.

15. The method of claim 9 , the method further comprising:
receiving, from the first wearable device, a subsequent manipulation of the virtual objects; and
communicating a visualization of the subsequent manipulation of the virtual objects to the second wearable device, causing the second wearable device to present the subsequent manipulation to the user designated as the expert.

16. The method of claim 9 , further comprising:
receiving video feeds from a plurality of wearable devices, the video feeds depicting the physical object;
generating a 3D model of the physical object based on the video feeds received from the plurality of wearable devices; and
upd

Assignees

• Daqri Llc

Inventors

• Mullins Brian
• Kammerait Matthew
• Broaddus Christopher

Classifications

• G06F3/1454 Primary

  involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay · CPC title

• G09G3/001

  using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G09G2370/16

  Use of wireless transmission of display information · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title