Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking

Patent metadata

Field	Value
Publication number	US-2025067980-A1
Application number	US-202418946168-A
Country	US
Kind code	A1
Filing date	Nov 13, 2024
Priority date	Feb 18, 2016
Publication date	Feb 27, 2025
Grant date	—

Abstract

A Head-Mounted Display system together with associated techniques for performing accurate and automatic inside-out positional, user body and environment tracking for virtual or mixed reality are disclosed. The system uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.

First claim

What is claimed is:

1 . A method comprising:
at a head-mounted device (HMD) including non-transitory memory, a first image sensor, one or more processors, and a display;
obtaining, via the first image sensor, an image of a physical environment;
obtaining a depth map associated with the physical environment;
generating, by using the depth map, a display image by blending the image of the physical environment with rendered graphics; and
displaying, via the display, the display image.

2 . The method of claim 1 , wherein the HMD includes a communication interface to communicate with a computer, wherein generating the display image is based at least in part on data from the computer via the communication interface.

3 . The method of claim 2 , wherein the data from the computer includes an encoded representation of the rendered graphics.

4 . The method of claim 3 , further comprising decoding at least a portion of the encoded representation of the rendered graphics to obtain the rendered graphics.

5 . The method of claim 2 , wherein the encoded representation of the rendered graphics indicates a plurality of color channels and an alpha mask.

6 . The method of claim 5 , wherein the alpha mask varies across the plurality of color channels based on display lens distortion characterizing the display.

7 . The method of claim 6 , wherein the display lens distortion includes color aberration of lenses of the display.

8 . The method of claim 5 , wherein blending the image of the physical environment with the rendered graphics includes performing alpha blending using the alpha mask.

9 . The method of claim 1 , further comprising generating the rendered graphics by performing embedded tracking based on the image of the physical environment and the depth map.

10 . The method of claim 9 , wherein performing the embedded tracking includes performing positional tracking of the HMD.

11 . The method of claim 10 , wherein the HMD includes an inertial measurement unit (IMU) that generates IMU data, wherein performing the positional tracking of the HMD is based on the IMU data.

12 . The method of claim 9 , wherein performing the embedded tracking includes performing body tracking of a user of the HMD or head tracking of the user of the HMD.

13 . The method of claim 9 , wherein the HMD includes a second image sensor, the method further comprising obtaining, via the second image sensor, one or more images of the physical environment, wherein performing embedded tracking is further based on the one or more images.

14 . The method of claim 1 , wherein the rendered graphics correspond to virtual content.

15 . A non-transitory computer-readable medium having instructions encoded thereon, which, when executed by one or more processors of a head-mounted device (HMD) including a first camera and a display, cause the HMD to:
obtain, via the first camera, an image of a physical environment; obtain a depth map associated with the physical environment; generate, by using the depth map, a display image by blending the image of the physical environment with rendered graphics; and display, via the display, the display image.

16 . The non-transitory computer-readable medium of claim 15 , wherein the HMD includes a communication interface to communicate with an computer, wherein generation of the display image is based at least in part on data from the computer via the communication interface, and wherein the data from the computer includes an encoded representation of the rendered graphics.

17 . The non-transitory computer-readable medium of claim 16 , wherein the encoded representation of the rendered graphics indicates a plurality of color channels and an alpha mask.

18 . The non-transitory computer-readable medium of claim 17 , wherein the alpha mask varies across the plurality of color channels based on display lens distortion characterizing the display.

19 . The non-transitory computer-readable medium of claim 17 , wherein blending the image of the physical environment with the rendered graphics includes performing alpha blending using the alpha mask.

20 . A head-mounted device (HMD) comprising:
a first image sensor to obtain an image of a physical environment; a depth sensor to obtain depth map associated with the physical environment; one or more processors to generate, by using the depth map, a display image by blending the image of the physical environment with rendered graphics; and a display to display the display image.

Assignees

• Apple Inc

Inventors

• Fortin-Deschenes Simon
• Chapdelaine-Couture Vincent
• Cote Yan
• Ghannoum Anthony

Classifications

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/0134

  of stereoscopic type · CPC title