Augmented reality systems and methods utilizing reflections

Patent metadata

Field	Value
Publication number	US-11244485-B2
Application number	US-201916543280-A
Country	US
Kind code	B2
Filing date	Aug 16, 2019
Priority date	Jan 19, 2016
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

A display system comprises a wearable display device for displaying augmented reality content. The display device comprises a display area comprising light redirecting features that are configured to direct light to a user. The display area is at least partially transparent and is configured to provide a view of an ambient environment through the display area. The display device is configured to determine that a reflection of the user is within the user's field of view through the display area. After making this determination, augmented reality content is displayed in the display area with the augmented reality content augmenting the user's view of the reflection. In some embodiments, the augmented reality content may overlie on the user's view of the reflection, thereby allowing all or portions of the reflection to appear to be modified to provide a realistic view of the user with various modifications made to their appearance.

First claim

What is claimed is:

1. A method for detecting a mirror in an environment, the method comprising:
under control of an augmented reality device (ARD) comprising computer hardware, the ARD comprising an outward-facing camera configured to image an environment around a user and an inertial measurement unit (IMU), the ARD configured to determine movements of a physical object in the environment,
collecting first movement data associated with the physical object, the first movement data being based on captured images of the physical object;
collecting second movement data associated with the user;
comparing the first movement data with the second movement data to identify information indicative of a presence of a mirror; and
confirming presence of the mirror based at least in part on the identified information.

2. The method of claim 1 , wherein the physical object is at least a portion of a body of the user, and wherein the images comprise reflections of the portion of the body of the user.

3. The method of claim 2 , wherein the second movement data associated with the user comprises movement data associated with a same portion of the user's body.

4. The method of claim 3 , wherein the portion of the body comprises a head or a hand.

5. The method of claim 1 , wherein collecting the first movement data and collecting the second movement data are performed by the outward-facing camera.

6. The method of claim 5 , wherein the first movement data comprises images indicating position changes of the physical object over a period of time and wherein the second movement data comprises images indicating position changes associated with the user over the same period of time.

7. The method of claim 1 , wherein collecting the first movement data is performed by the outward-facing camera and collecting the second movement data is performed by the IMU.

8. The method of claim 7 , wherein the first movement data comprises images indicating position changes of the physical object over a period of time and wherein the second movement data comprises position changes associated with the user over the period of time.

9. The method of claim 1 , wherein comparing the first movement data with the second movement data to identify the information comprises:
generating a first trajectory associated with the physical object, the first trajectory based at least partly on the first movement data;
generating a second trajectory associated with the user, the second trajectory based at least partly on the second movement data; and
calculating the information indicative of a presence of a mirror, the information comprising a covariance of the first trajectory and the second trajectory.

10. The method of claim 9 , wherein confirming presence of the mirror is based on the covariance being less than a threshold.

11. The method of claim 9 , wherein the physical object comprises a portion of a body, and wherein the second trajectory associated with the user comprises a trajectory associated with a same portion of the user's body.

12. A method for detecting a reflective surface in an environment, the method comprising:
under control of an augmented reality device (ARD) comprising computer hardware, the ARD comprising an outward-facing camera configured to image an environment around a user and an inertial measurement unit (IMU), the ARD configured to track movements of a physical object in the environment,
collecting first movement data associated with the physical object;
collecting second movement data associated with the user; and
comparing the first movement data with the second movement data to identify information indicative of a presence of a mirror.

13. The method of claim 12 , wherein the physical object comprises at least a portion of a body of the user, wherein the first movement data is based on images of the physical object, and wherein the images of the physical object comprise reflections of the portion of the body of the user.

14. The method of claim 13 , wherein the physical object comprises a head or a hand of the user, and wherein the second movement data associated with the user comprises movement data associated with the head or the hand of the user.

15. The method of claim 12 , wherein collecting the first movement data is performed by the outward-facing camera and collecting the second movement data is performed by the IMU.

16. The method of claim 12 , wherein collecting the first movement data and collecting the second movement data comprise:
obtaining images, by the outward-facing camera, over a period of time;
identifying respective positions of the physical object in the images;
calculating a first trajectory based on the identified respective positions of the physical object;
identifying respective positions associated with the user over the period of time; and
calculating a second trajectory based on the identified respective positions associated with the user.

17. The method of claim 16 , wherein identifying the information indicative of a presence of a mirror comprises:
calculating a covariance between the first movement data and the second movement data.

18. The method of claim 17 , wherein the mirror is identified based on the covariance being less than a threshold.

19. The method of claim 16 , wherein the physical object comprises a portion of a body, and wherein the positions associated with the user comprise positions associated with a same portion of the user's body.

20. An augmented reality (AR) system for detecting a reflective surface in an environment, the AR system comprising:
one or more outwardly-facing cameras configured to image an environment around a user;
an inertial measurement unit configured to measure the movement of a portion of the user;
hardware processors; and
non-transitory computer storage media storing instructions that when executed by the one or more hardware processors, cause the one or more processors to perform operations comprising:
collecting first movement data associated with a physical object in the environment, the first movement data being based on images of the physical object;
collecting second movement data associated with the user;
comparing the first movement data with the second movement data to identify information indicative of a presence of a mirror; and
confirming presence of the mirror based at least in part on the identified information.

21. The AR system of claim 20 , wherein the physical object is at least a portion of a body of the user, and wherein the images comprise reflections of the portion of the body of the user.

22. The AR system of claim 21 , wherein second movement data associated with the user comprises movement data associated with a portion of the user's body corresponding to the portion of the body in the reflections.

23. The AR system of claim 20 , wherein collecting the first movement data and collecting the second movement data are performed by the one or more outwardly-facing cameras.

24. The AR system of claim 23 , wherein the first movement data comprises images indicating position changes of the physical object over a period of time, and wherein the second movement data comprises images indicating position changes associated with the user over the period of time.

25. The AR system of claim 20 , wherein collecting the first movement data is performed by the outwardly-facing camera and collecting the second movement data is p

Assignees

• Magic Leap Inc

Inventors

• Harrises Christopher M
• Samec Nicole Elizabeth
• Robaina Nastasja U
• Baerenrodt Mark
• Wright Adam Carl
• Kaehler Adrian

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G02B27/017 Primary

  Head mounted · CPC title

• G02B27/0101 Primary

  characterised by optical features (G02B27/0172 takes precedence) · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title