Mixed reality data collaboration

Patent metadata

Field	Value
Publication number	US-9685003-B2
Application number	US-201313908816-A
Country	US
Kind code	B2
Filing date	Jun 3, 2013
Priority date	Jun 3, 2013
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

Embodiments that relate to sharing mixed reality experiences among multiple display devices are disclosed. In one embodiment, a method includes receiving current versions of a plurality of data subtypes geo-located at a keyframe location. A world map data structure is updated to include the current versions, and a neighborhood request including the keyframe location is received from a display device. Based on the keyframe location, an identifier and current version indicator for each data subtype is provided to the device. A data request from the device for two or more of the data subtypes is received, and the two or more data subtypes are prioritized based on a priority hierarchy. Based on the prioritization, current versions of the data subtypes are sequentially provided to the device for augmenting an appearance of a mixed reality environment.

First claim

The invention claimed is:

1. A data collaborator for enabling two or more display devices to share mixed reality experiences, the data collaborator comprising:
a processor configured to:
receive a plurality of data subtypes from a first display device, each of the data subtypes being geo-located at a keyframe location, having a current version, and including physical environment data and virtual environment data;
update a world map data structure to include the current versions of the data subtypes at the keyframe location;
receive a neighborhood request including the keyframe location from a second display device;
in response to the neighborhood request and based on the keyframe location, provide to the second display device an identifier for each of the plurality of data subtypes that identifies a subtype of each of the plurality of data subtypes and a current version indicator for each of the plurality of data subtypes, and not provide any of the plurality of data subtypes to the second display device;
receive a data request from the second display device for two or more of the plurality of data subtypes, the data request including the identifier for each of the two or more data subtypes;
prioritize the two or more data subtypes based on a priority hierarchy, the priority hierarchy prioritizing one of a transform data subtype and an image data subtype above the other of the transform data subtype and the image data subtype; and
based on the prioritization, sequentially provide current versions of the two or more data subtypes to the second display device such that a first data subtype is provided to the second display device before a second data subtype, wherein an appearance of a mixed reality environment generated via the second display device is augmented using the current versions of the two or more data subtypes.

2. The data collaborator of claim 1 , wherein the data request includes a non-current version indicator for each of the two or more data subtypes.

3. The data collaborator of claim 2 , wherein the processor is further configured to:
compare the non-current version indicators to the current version indicators for the two or more data subtypes; and
based on the comparison, select the current versions of the two or more data subtypes for provision to the second display device.

4. The data collaborator of claim 1 , wherein the plurality of data subtypes are prioritized by the first display device based on the priority hierarchy, and the plurality of data subtypes are sequentially received from the first display device based on the priority hierarchy.

5. The data collaborator of claim 1 , wherein the plurality of data subtypes are prioritized by the second display device based on the priority hierarchy, and the data request from the second display device includes a prioritization of the two or more data subtypes based on the priority hierarchy.

6. The data collaborator of claim 1 , wherein prior to receiving the neighborhood request from the second display device and in response to receiving the plurality of data subtypes from the first display device, the processor is further configured to provide to the second display device the identifiers that identify each of the plurality of data subtypes and the current version of each of the plurality of data subtypes received from the first display device.

7. The data collaborator of claim 1 , wherein the processor is further configured to:
determine that a previous version of one of the two or more data subtypes was previously received from the second display device; and
based on this determination, refrain from providing the previous version of the one of the two or more data subtypes to the second display device.

8. The data collaborator of claim 1 , wherein the data subtypes are selected from the group consisting of pose link transforms, anchor transforms, tracking images, scene images, correspondences between two-dimensional image coordinates and three-dimensional locations, keyframe metadata, keyframe descriptors, three-dimensional keyframe relocalization points, quantizations of keyframe descriptors, and rig-to-rig transforms.

9. The data collaborator of claim 1 , wherein the priority hierarchy comprises a first priority that includes anchor transforms, a second priority that includes scene images, and a third priority that includes three-dimensional keyframe relocalization points.

10. The data collaborator of claim 1 , wherein the second display device is a head-mounted display device.

11. A method for enabling two or more display devices to share mixed reality experiences, the method comprising:
receiving a plurality of data subtypes from a first display device, each of the data subtypes being geo-located at a keyframe location, having a current version, and including physical environment data and virtual environment data;
updating a world map data structure to include the current versions of the data subtypes at the keyframe location;
receiving a neighborhood request including the keyframe location from a second display device;
in response to the neighborhood request and based on the keyframe location, providing to the second display device an identifier for each of the plurality of data subtypes that identifies a subtype of each of the plurality of data subtypes and a current version indicator for each of the plurality of data subtypes, and not providing any of the plurality of data subtypes to the second display device;
receiving a data request from the second display device for two or more of the plurality of data subtypes, the data request including the identifier for each of the two or more data subtypes;
prioritizing the two or more data subtypes based on a priority hierarchy, the priority hierarchy prioritizing one of a transform data subtype and an image data subtype above the other of the transform data subtype and the image data subtype; and
based on the prioritization, sequentially providing current versions of the two or more data subtypes to the second display device such that a first data subtype is provided to the second display device before a second data subtype, wherein an appearance of a mixed reality environment generated via the second display device is augmented using the current versions of the two or more data subtypes.

12. The method of claim 11 , wherein the data request includes a non-current version indicator for each of the two or more data subtypes.

13. The method of claim 12 , further comprising:
comparing the non-current version indicators to the current version indicators for the two or more data subtypes; and
based on the comparison, selecting the current versions of the two or more data subtypes for provision to the second display device.

14. The method of claim 11 , wherein the plurality of data subtypes are prioritized by the first display device based on the priority hierarchy, and the method further comprises sequentially receiving the plurality of data subtypes from the first display device based on the priority hierarchy.

15. The method of claim 11 , wherein the plurality of data subtypes are prioritized by the second display device based on the priority hierarchy, and the data request from the second display device includes a prioritization of the two or more data subtypes based on the priority hierarchy.

16. The method of claim 11 , further comprising, prior to receiving the neighborhood request from the second display device and in response to receiving the plurality of data subtypes from the first display device, providing to the second display device the identifiers that

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lazarow Gavin
• Schaffalitzky Frederik
• Hou Dan

Classifications

• G09G2370/022

  Centralised management of display operation, e.g. in a server instead of locally · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G02B27/0172

  characterised by optical features · CPC title