Recognizing objects in a passable world model in an augmented or virtual reality system

Patent metadata

Field	Value
Publication number	US-9767616-B2
Application number	US-201514706994-A
Country	US
Kind code	B2
Filing date	May 8, 2015
Priority date	Apr 18, 2014
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

First claim

What is claimed is:

1. A method for recognizing objects with a virtual or augmented reality display system, comprising:
identifying map points in a world model that is projected to at least one eye of a user by a projector in a virtual or augmented reality display system, wherein the map points corresponds to one or more real objects and is used to determine a geometric structure of a location of the user;
recognizing, using at least one microprocessor, a real object of the one or more real objects in the world model presented by the virtual or augmented reality display system to the user at least by deriving a geometric structure of the real object using an image segmentation technique based in part or in whole upon the map points; and
displaying one or more virtual objects to the user relative to the real object, based at least in part on a location of the user.

2. The method of claim 1 , further comprising:
determining a set of parameters corresponding to a movement of the user of the augmented reality system relative to the detected location;
calculating, based on the determined movement of the user, an anticipated position of the user; and
retrieving another data pertaining to the anticipated position of the user, wherein the other data pertaining to the anticipated position comprises map points corresponding to one or more real objects of the anticipated position.

3. The method of claim 1 , wherein the map points corresponding to one or more real objects are used to construct a map of the real world.

4. The method of claim 1 , further comprising recognizing one or more objects of the real world based in part or in whole on the map points.

5. The method of claim 1 , wherein the map points are used to create a coordinate space of the real world, and wherein the one or more virtual objects are displayed based on the created coordinate space of the real world.

6. The method of claim 1 , further comprising:
recognizing one or more objects of the real world based in part or in whole on the map points; and
displaying the virtual object based at least in part on a property of the one or more objects that have been recognized.

7. The method of claim 1 , wherein the map points pertain to a geometry of the location.

8. The method of claim 1 , further comprising downloading a set of map points pertaining to the location to the augmented reality display system.

9. The method of claim 1 , further comprising uploading another set of map points from the augmented reality display system to a cloud resource.

10. The method of claim 1 , recognizing the real object further comprising: for
capturing, at one or more sensors of the virtual or augmented reality display system, a plurality of pose tagged images of surroundings of the user;
extracting a set of points from the plurality of pose tagged images;
segmenting, using at least one microprocessor, image data of a real object of the one or more real objects into segmented object features; and
correlating the segmented object features with the set of points.

11. The method of claim 10 , further comprising:
deriving the geometric structure of using at least the segmented object features correlated with the set of points; and
recognizing a real object of the or more real objects using the geometric structure and an object recognizer of a plurality of object recognizers.

12. The method of claim 11 , further comprising:
parameterizing the geometric structure of the real object into parametric primitive for the real object; and
attaching semantic information to the parametric primitive.

13. The method of claim 12 , further comprising:
synchronizing the parametric primitive of the real object to the world model; and
estimating a future position of the real object in the world model using at least the semantic information, wherein the semantic information includes a taxonomical descriptor.

14. The method of claim 1 , further comprising:
identifying a first real object of a first type;
recognizing the real object of the first type with a first object recognizer that recognizes the first type of object; and
recognizing a smaller subset of the real object of the first type with a second object recognizer.

15. The method of claim 1 , identifying the map points comprising:
identifying two-dimensional (2D) map points captured with one or more respective sensors of multiple virtual or augmented reality display systems;
generating a geometric map comprising a keyframe that includes the map points, wherein the geometric map includes at least two nodes connected with an edge that corresponds to a number of shared map points between the at least two nodes; and
identifying a point of stress in the geometric map based in part or in whole upon information in a topological map, a discrepancy in a location of the keyframe, in relation to the geometric map, or a residual error of the geometric map.

16. The method of claim 15 , identifying the map points comprising:
identifying a first node based in part or in whole upon the point of stress;
identifying a first wave of nodes having a single degree of separation from the first node; and
distributing the point of stress to the first wave of nodes.

17. The method of claim 1 , further comprising:
identifying the map points pertaining to a real world environment;
determining a position of a plurality of keyframes that are used to determine the map points; and
determining at least one new map point based in part or in whole upon the map points and the position of the plurality of keyframes that are used to determine the map points.

18. The method of claim 17 , further comprising:
rendering respective triangular cones from the position of the plurality of keyframes to at least some of the map points at least by placing a respective map point along a bisector of a corresponding triangular cone;
selectively shading the respective triangular cones such each bisector of a corresponding triangular cone is associated with a graphical emphasis in relation to a remaining portion of the corresponding triangular cone;
identifying a plurality of points of intersection between two intersecting triangular cones; and
identifying the at least one new map point based in part or in whole upon the graphical emphases corresponding to the plurality of points of intersection or upon a pixel pitch corresponding to the plurality of points of intersection.

19. The method of claim 17 , further comprising:
placing a virtual keyframe in relation to the plurality of keyframes;
determining a relatively more orthogonal direction to the plurality of keyframes;
positioning the virtual keyframe along the relatively more orthogonal direction;
rendering respective lines from the virtual keyframe to the map points;
determining a plurality of points of intersections among the respective lines; and
determining the at least one new map point based in part or in whole upon the plurality of points of intersection rendered from the virtual keyframe to the map points.

20. A system for displaying augmented reality, comprising:
a virtual or augmented reality display system configured to identify map points in a world model that is presented to a user, wherein the map points corresponds to one or more real objects and is used to determine a geometric structure of a location of the user;
at least one microprocessor configured to recognize a real object of the one or more real objects in the world model presented by the virtual or augmented re

Assignees

• Magic Leap Inc

Inventors

• Miller Samuel A

Classifications

• H04S2400/15

  Aspects of sound capture and related signal processing for recording or reproduction · CPC title

• H04S2400/11

  Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) · CPC title

• H04S7/304

  For headphones · CPC title

• G06Q30/02

  Marketing; Price estimation or determination; Fundraising · CPC title

• A63F13/577

  using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars · CPC title