Augmented reality systems and methods utilizing reflections

Patent metadata

Field	Value
Publication number	US-12260476-B2
Application number	US-202217590724-A
Country	US
Kind code	B2
Filing date	Feb 1, 2022
Priority date	Jan 19, 2016
Publication date	Mar 25, 2025
Grant date	Mar 25, 2025

Abstract

A display system comprises a wearable display device for displaying augmented reality content. The display device comprises a display area comprising light redirecting features that are configured to direct light to a user. The display area is at least partially transparent and is configured to provide a view of an ambient environment through the display area. The display device is configured to determine that a reflection of the user is within the user's field of view through the display area. After making this determination, augmented reality content is displayed in the display area with the augmented reality content augmenting the user's view of the reflection. In some embodiments, the augmented reality content may overlie on the user's view of the reflection, thereby allowing all or portions of the reflection to appear to be modified to provide a realistic view of the user with various modifications made to their appearance.

First claim

What is claimed is:

1. A method for sharing an image in an augmented reality telepresence session, the method comprising:
establishing a bidirectional communication between a first augmented reality device associated with a first caller and a second augmented reality device associated with a second caller, the first augmented reality device comprising a first outward facing camera configured to image a region in front of the first augmented reality device;
detecting presence of a reflective surface in a region in front of the first augmented reality device, wherein the detecting the presence of the reflective surface includes keypoints recognition of characteristics of the region in front of the first augmented reality device and one or more depth discontinuities within the reflective surface, wherein detecting the presence of the reflective surface comprises:
identifying a first set of keypoints based on the keypoints recognition of characteristics of the region in front of the first augmented reality device, the first set of keypoints including a first triplet of neighboring keypoints;
maintaining a map of an environment around the first augmented reality device, the map comprising a second set of keypoints corresponding to objects in the environment, the second set of keypoints including a second triplet of neighboring keypoints;
determining geometric relationships between the first triplet of neighboring keypoints and between the second triplet of neighboring keypoints; and
matching the geometric relationships associated with the first triplet of neighboring keypoints and the second triplet of neighboring keypoints;
confirming the presence of the reflective surface based on the matching of the geometric relationships;
capturing, with the first outward facing camera, a first image of the first caller reflected from the reflective surface;
identifying a region of the first image of the first caller that includes an image of the first augmented reality device occluding a portion of a face of the first caller;
generating an unoccluded image of the first caller by substituting the identified region with a corresponding region of an image of the unoccluded face of the first caller without the image of the first augmented reality device that previously occluded the portion of the face of the first caller;
transmitting at least a portion of the unoccluded image of the first caller to the second augmented reality device; and
the determining the geometric relationships between the first triplet of neighboring keypoints and the second triplet of neighboring keypoints includes at least one of a) determining open angles among the first triplet of neighboring keypoints and open angles among the second triplet of neighboring keypoints, b) axially transforming a first geometry represented by the first triplet of neighboring keypoints or axially transforming a second geometry represented by the first triplet of neighboring keypoints, or c) determining an amount of magnification between a first size of the first geometry and a second size of the second geometry.

2. The method of claim 1 , further comprising:
receiving, by the first augmented reality device, a second image of the second caller; and
displaying at least a portion of the second image of the second caller by the first augmented reality device.

3. The method of claim 2 , wherein displaying at least a portion of the second image of the second caller comprises displaying the second image so that it occludes the first image of the first caller in the reflective surface.

4. The method of claim 1 , wherein transmitting the portion of the unoccluded image of the first caller to the second augment reality device comprises:
transmitting the unoccluded image of the first caller to a server computer system;
processing the unoccluded image of the first caller by the server computer system; and
transmitting the unoccluded image of the first caller to the second augmented reality device.

5. The method of claim 1 , further comprising processing the first image of the first caller by performing at least one of the following:
applying virtual content to the first image of the first caller; cropping a portion of the first image of the first caller; applying a background to the first image of the first caller; resizing the first image of the first caller; and
identifying a region of a body of the first caller that is not captured by the first augmented reality device and adding the identified region to the first image of the first caller.

6. The method of claim 5 , wherein the virtual content comprises blending effects, feathering effects, glow effects, or special features effects.

7. The method of claim 5 , wherein the background contains elements associated with games, movies, books, art, music, existing physical elements, or virtual objects.

8. The method of claim 1 , wherein the first augmented reality device comprises an eye-tracking camera configured to image an eye of the first caller, and wherein substituting the identified region with the corresponding region of the image of the unoccluded face of the first caller comprises inserting an image of the eye of the first caller in the identified region.

9. The method of claim 1 , wherein the unoccluded image of the first caller comprises an image patch obtained before the augmented reality telepresence session.

10. The method of claim 1 , wherein generating the unoccluded image of the first caller comprises dynamically deforming an image patch of the unoccluded face of the first caller based on data obtained from an inward-facing imaging system of the first augmented reality device.

11. The method of claim 1 , wherein the establishing the bidirectional communication connection automatically initiated in response to detecting the presence of the reflective surface.

12. An augmented reality system for conducting a teleconference between a first caller and a second caller, the system comprising:
a first augmented reality device associated with a first caller, the first augmented reality device comprising a first outward facing camera configured to image a region in front of the first augmented reality device; and
a computer processor configured to communicate with a network and programmed to:
establish a bidirectional communication between the first augmented reality device and a second augmented reality device associated with a second caller;
detect presence of a reflective surface in a region in front of the first augmented reality device, including: maintaining a map of an environment around the first augmented reality device, the map comprising a first set of keypoints corresponding to objects in the environment; identifying a second set of keypoints in the images captured by the first outward facing camera; determining geometric relationships between the first set of keypoints and between the second set of keypoints; matching the geometric relationships associated with the first set of keypoints and the second set of keypoints; and confirming the presence of the reflective surface based on the matching of the geometric relationships, wherein the first set of keypoints includes a first triplet of neighboring keypoints and the second set of keypoints includes a second triplet of neighboring keypoints,
wherein the computer processor is programmed to detect the presence of the reflective surface, including: determine the geometric relationships between the first set of keypoints and between the second set of keypoints by determining the geometric relationships between the first triplet of neighboring keypoints and between the second triplet of neighboring keypoints; and match

Assignees

• Magic Leap Inc

Inventors

• Harrises Christopher M
• Samec Nicole Elizabeth
• Robaina Nastasja U
• Baerenrodt Mark
• Wright Adam Carl
• Kaehler Adrian

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06V20/20

  in augmented reality scenes · CPC title