Capturing augmented reality on a head mounted display

Patent metadata

Field	Value
Publication number	US-11508127-B2
Application number	US-201816190037-A
Country	US
Kind code	B2
Filing date	Nov 13, 2018
Priority date	Nov 13, 2018
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

To capture augmented reality (AR) on a head mounted display the embodiments described herein include the optical display of virtual components of an AR on an AR device. A real world view within a field of view of the display of the AR device is captured, creating a composite image that combines the virtual components and real world physical components within the field of view of the AR device.

First claim

What is claimed is:

1. A method, comprising:
capturing, using a visual sensor fixed within an augmented reality (AR) device and facing a display of the AR device, an AR environment image,
wherein the AR environment image comprises at least one virtual component and a user environment background,
wherein the at least one virtual component and the user environment background are simultaneously captured by the visual sensor,
wherein the user environment background is a real world view within a field of view of the display,
wherein the visual sensor is positioned within the AR device and between a center of a line of sight of an eye of a user of the AR device and the display of the AR device, wherein the visual sensor is positioned in the AR device to match a position of the eye of the user to prevent offset correction image processing prior to external transmission, wherein the AR environment image comprises a field of view of the display as viewed by the eye of the user within the AR device,
wherein a light associated with the user environment background and captured by the visual sensor passes directly through the display of the AR device to the visual sensor, and
wherein the at least one virtual component is reflected on the display of the AR device; and
processing the AR environment image for external transmission.

2. The method of claim 1 , wherein processing the AR environment image further comprises:
determining, using the visual sensor, a user environment layout; and
generating a virtual background based on the user environment layout; and
combining the at least one virtual component reflected on the display of the AR device with the virtual background to create the AR environment image for external transmission.

3. The method of claim 1 , wherein capturing the AR environment image comprises:
capturing a user environment image, using the visual sensor, wherein the visual sensor is affixed to the AR device; and
combining the at least one virtual component with the user environment image to generate the AR environment image for external transmission.

4. The method of claim 1 , wherein the AR device comprises an all in one head mounted AR device.

5. The method of claim 1 , wherein the visual sensor is positioned within a line of sight between the user of the AR device and the display of the AR device.

6. The method of claim 1 , wherein the display of AR device comprises a beam splitter configured to combine the AR with the real world view by directing the at least one virtual component onto the display of the AR device.

7. The method of claim 1 , wherein processing the AR environment image for external transmission further comprises:
determining a position of the eye of the user to determine a focus point of the user within the field of view of the display; and
wherein processing the AR environment image for external transmission comprises:
processing the AR environment image to focus on the focus point of the user.

8. The method of claim 1 , further comprising:
capturing an user environment ambient audio corresponding to a real world sound;
combining the user environment ambient audio with AR audio into a combined audio stream; and
transmitting the combined audio and the AR environment image as a stream.

9. A computer-readable storage medium comprising computer-readable program code embodied therewith, the computer-readable program code is configured to perform, when executed by a processor, an operation, the operation comprising:
capturing, using a visual sensor fixed within an augmented reality (AR) device and facing a display of the AR device, an AR environment image,
wherein the AR environment image comprises at least one virtual component and a user environment background,
wherein the at least one virtual component and the user environment background are simultaneously captured by the visual sensor,
wherein the user environment background is a real world view within a field of view of the display,
wherein the visual sensor is positioned within the AR device and between a center of a line of sight of an eye of a user of the AR device and the display of the AR device, wherein the visual sensor is positioned in the AR device to match a position of the eye of the user to prevent offset correction image processing prior to external transmission, wherein the AR environment image comprises a field of view of the display as viewed by the eye of the user within the AR device,
wherein a light associated with the user environment background and captured by the visual sensor passes directly through the display of the AR device to the visual sensor, and
wherein the at least one virtual component is reflected on the display of the AR device; and

processing the AR environment image for external transmission.

10. The computer-readable storage medium of claim 9 , wherein determining the user environment background comprises:
determining, using the visual sensor, a user environment layout; and
generating a virtual background based on the user environment layout; and
combining the at least one virtual component reflected on the display of the AR device with the virtual background to create the AR environment image for external transmission.

11. The computer-readable storage medium of claim 9 , wherein determining the user environment background comprises:
capturing a user environment image, using the visual sensor, wherein the visual sensor is affixed to the AR device; and
combining the at least one virtual component with the user environment image to generate the AR environment image for external transmission.

12. The computer-readable storage medium of claim 9 , wherein the visual sensor is positioned within a line of sight between the user of the AR device and the display of the AR device.

13. The computer-readable storage medium of claim 9 , wherein the display of AR device comprises a beam splitter configured to combine the AR with the real world view by directing the at least one virtual component onto the display of the AR device.

14. The computer-readable storage medium of claim 9 , wherein the operation further comprises:
determining a position of the eye of the user to determine a focus point of the user within the field of view of the display; and
wherein processing the AR environment image for external transmission comprises:
processing the AR environment image to focus on the focus point of the user.

15. A system, comprising:
a processor; and
a memory comprising instructions which, when executed on the processor, performs an operation, the operation comprising:
capturing, using a visual sensor fixed within an augmented reality (AR) device and facing a display of the AR device, an AR environment image,
wherein the AR environment image comprises at least one virtual component and a user environment background,
wherein the at least one virtual component and the user environment background are simultaneously captured by the visual sensor,
wherein the user environment background is a real world view within a field of view of the display,
wherein the visual sensor is positioned within the AR device and between a center of a line of sight of an eye of a user of the AR device and the display of the AR device, wherein the visual sensor is positioned in the AR device to match a position of the eye of the user to prevent offset correction image processing prior to external transmission, wherein the AR environment image comprises a field of view of the display as viewed by the eye of the user within the AR device,
wherein a light associated with the user environment

Assignees

• Disney Entpr Inc

Inventors

• Goslin Michael P
• Nocon Nathan D
• Hsu Jonathan R D
• Haseltine Eric C
• Baumbach Elliott H

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/017

  Head mounted · CPC title