Matching content to a spatial 3D environment

Patent metadata

Field	Value
Publication number	US-10930076-B2
Application number	US-201815968673-A
Country	US
Kind code	B2
Filing date	May 1, 2018
Priority date	May 1, 2017
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

Systems and methods for matching content elements to surfaces in a spatially organized 3D environment. The method includes receiving content, identifying one or more elements in the content, determining one or more surfaces, matching the one or more elements to the one or more surfaces, and displaying the one or more elements as virtual content onto the one or more surfaces.

First claim

What is claimed is:

1. A method comprising:
receiving content;
identifying a plurality of elements in the content, wherein each of the plurality of elements is a subset of the content;
determining a plurality of surfaces;
matching the plurality of elements to the plurality of surfaces; and
displaying the plurality of elements as virtual content onto the plurality of surfaces.

2. The method of claim 1 , wherein the content comprises at least one of a pulled content or pushed content.

3. The method of claim 1 , wherein identifying the plurality of elements comprises parsing the content to identify the plurality of elements.

4. The method of claim 1 , wherein identifying the plurality of elements comprises determining one or more attributes for each of the plurality of elements.

5. The method of claim 4 , wherein the one or more attributes include at least one of a priority attribute, an orientation attribute, an aspect ratio attribute, a dimension attribute, an area attribute, a relative viewing position attribute, a color attribute, a contrast attribute, a position type attribute, a margin attribute, a type of content attribute, a focus attribute, a readability index attribute, or a type of surface to place attribute.

6. The method of claim 4 , wherein determining the one or more attributes for each of the plurality of elements is based on explicit indications in the content.

7. The method of claim 4 , wherein determining the one or more attributes for each of the plurality of elements is based on placement of the plurality of elements within the content.

8. The method of claim 1 , further comprising storing the plurality of elements into one or more logical structures.

9. The method of claim 8 , wherein the one or more logical structures comprise at least one of an ordered array, a hierarchical table, a tree structure, or a logical graph structure.

10. The method of claim 1 , wherein the plurality of surfaces comprise at least one of physical surfaces or virtual surfaces.

11. The method of claim 1 , wherein determining the plurality of surfaces comprises parsing an environment to determine at least one of the plurality of e surfaces.

12. The method of claim 1 , wherein determining the plurality of surfaces comprises:
receiving raw sensor data;
simplifying the raw sensor data to produce simplified data; and
creating one or more virtual surfaces based on the simplified data;
wherein the plurality of surfaces comprise the one or more virtual surfaces.

13. The method of claim 12 , wherein simplifying the raw sensor data comprises:
filtering the raw sensor data to produce filtered data; and
grouping the filtered data into one or more groups by point cloud points;
wherein the simplified data includes the one or more groups.

14. The method of claim 13 , wherein creating the one or more virtual surfaces comprises:
iterating through each of the plurality of groups to determine one or more real world surfaces; and
creating the one or more virtual surfaces based on the one or more real world surfaces.

15. The method of claim 1 , wherein determining the plurality of surfaces comprises determining one or more attributes for each of the plurality of surfaces.

16. The method of claim 15 , wherein the one or more attributes include at least one of a priority attribute, an orientation attribute, an aspect ratio attribute, a dimension attribute, an area attribute, a relative viewing position attribute, a color attribute, a contrast attribute, a position type attribute, a margin attribute, a type of content attribute, a focus attribute, a readability index attribute, or a type of surface to place attribute.

17. The method of claim 1 , further comprising storing the plurality of surfaces into one or more logical structures.

18. The method of claim 1 , wherein matching the plurality of elements to the plurality of surfaces comprises:
prioritizing the plurality of elements;
for each element of the plurality of elements:
comparing one or more attributes of the element to one or more attributes of each of the plurality of surfaces;
calculating a match score based on the one or more attributes of the element and the one or more attributes of each of the plurality of surfaces; and
identifying a best matching surface having a highest match score.

19. The method of claim 18 , further comprising:
for each of the plurality of elements:
storing an association between the element and the best matching surface.

20. The method of claim 1 , wherein one element is matched to one or more surfaces.

21. The method of claim 20 , further comprising:
displaying each surface of the plurality of surfaces to a user;
receiving a user selection indicating a winning surface from the plurality of surfaces displayed; and
saving, from the user selection, surface attributes of the winning surface in a user preferences data structure.

22. The method of claim 1 , wherein the content is data streamed from a content provider.

23. The method of claim 1 , wherein the plurality of elements are displayed to a user through a mixed reality device.

24. The method of claim 1 , further comprising displaying one or more additional surface options for displaying the plurality of elements based at least in part on a changed field of view of a user.

25. The method of claim 24 , wherein the displaying of the one or more additional surface options is based at least in part on a time threshold corresponding to the changed field of view.

26. The method of claim 24 , wherein the displaying of the one or more additional surface options is based at least in part on a headpose change threshold.

27. The method of claim 1 , further comprising overriding displaying the plurality of elements onto the plurality of surfaces that were matched.

28. The method of claim 27 , wherein overriding the displaying of the plurality of elements onto the plurality of surfaces is based at least in part on historically frequently used surfaces.

29. The method of claim 1 , further comprising moving the plurality of elements displayed on the plurality of surfaces to a different surface based at least in part on a user selecting a particular element displayed at the plurality of surfaces to be moved to the different surface.

30. The method of claim 29 , wherein the particular element is moved to the different surface is at least viewable by the user.

31. The method of claim 1 , further comprising in response to a change to a field of view of a user from a first field of view to a second field of view, moving the displaying of the plurality of elements onto new surfaces to follow the change of the user's field of view to the second field of view,
wherein the displaying of the plurality of elements is moved with small discrete incremental positions between the plurality of surfaces and the new surfaces.

32. The method of claim 31 , wherein the plurality of elements may only move directly in front of the user's second field of view upon confirmation from received from the user to move the content directly in front of the user's second field of view.

33. The method of claim 1 , further comprising pausing displaying of the plurality of elements onto the plurality of surfaces at a first location and resuming displaying of t

Assignees

• Magic Leap Inc

Inventors

• Bastov Denys
• Ng-Thow-Hing Victor
• Reinhardt Benjamin Zaaron
• ZOLOTAREV Leonid
• Pellet Yannick
• Marchenko Aleksei
• Meaney Brian Everett
• Shelton Marc Coleman
• Geiman Megan Ann
• Gotcher John A
• Bogue Matthew Schon
• Balasubramanyam Shivakumar
• Ruediger Jeffrey Edward
• Lundmark David Charles

Classifications

• G02B30/00

  Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images (in microscopes G02B21/22) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• H04N21/816

  involving special video data, e.g 3D video · CPC title