Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking

Patent metadata

Field	Value
Publication number	US-10838206-B2
Application number	US-201716063004-A
Country	US
Kind code	B2
Filing date	Feb 20, 2017
Priority date	Feb 18, 2016
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

A Head-Mounted Display system together with associated techniques for performing accurate and automatic inside-out positional, user body and environment tracking for virtual or mixed reality are disclosed. The system uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.

First claim

We claim:

1. A method comprising:
at a head-mounted device including an image sensor, a display, a communications interface, and one or more processors:
capturing, via the image sensor, an image of a scene;
obtaining, via the communications interface, an image of content to be displayed in association with the image of the scene;
obtaining a depth map for at least a portion of the scene;
performing, using the one or more processors, one or more image signal processing functions on the image of the scene;
determining an alpha mask for the image of content by comparing depth values for each pixel in the image of content to depth vales for pixels in the processed image of the scene based on the depth map for at least the portion of the scene;
generating a display image by combining, using the one or more processors, the image of the content with the processed image of the scene based at least in part on the alpha mask; and
displaying, on the display, the display image.

2. The method of claim 1 , wherein displaying the display image is performed within 20 milliseconds of capturing the image of the scene.

3. The method of claim 1 , wherein the image of the scene is not transmitted via the communications interface.

4. The method of claim 1 , wherein a representation of the image of the scene is transmitted via the communications interface to enable remote computer vision processing.

5. The method of claim 1 , wherein neither the image of the scene, the processed image of the scene, nor the display image is received via the communications interface.

6. The method of claim 1 , wherein the head-mounted device includes one or more positional tracking sensors generating positional tracking data, further comprising, transmitting, via the communications interface, the positional tracking data, wherein the image of the content is based on the positional tracking data.

7. The method of claim 1 , wherein the one or more image signal processing functions include correction of image distortion for the display.

8. The method of claim 1 , wherein the one or more image signal processing functions include one or more of debayering, color correction, or noise reduction.

9. A head-mounted device comprising:
an image sensor to capture an image of a scene;
a depth sensor to obtain a depth map for at least a portion of the scene;
a communications interface to obtain an image of content to be displayed in association with the image of the scene; and
one or more processors to:
perform one or more image signal processing functions on the image of the scene; and
determine an alpha mask for the image of content by comparing depth values for each pixel in the image of content to depth vales for pixels in the processed image of the scene based on the depth map for at least the portion of the scene
generate a display image by combining the image of the content with the processed image of the scene based at least in part on the alpha mask; and
a display to display the display image.

10. The head-mounted device of claim 9 , wherein the display is to display the display image within 20 milliseconds of the image sensor capturing the image of the scene.

11. The head-mounted device of claim 9 , wherein the image of the scene is not transmitted via the communications interface.

12. The head-mounted device of claim 9 , wherein neither the image of the scene, the processed image of the scene, nor the display image is received via the communications interface.

13. The head-mounted device of claim 9 , wherein the one or more image signal processing functions include correction of image distortion for the display.

14. The head-mounted device of claim 9 , wherein the one or more image signal processing functions include one or more of debayering, color correction, or noise reduction.

15. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by one or more processors of a head-mounted device including an image sensor, a communications interface, and a display caused the head-mounted device to:
capture, via the image sensor, an image of a scene;
obtain, via the communications interface, an image of content to be displayed in association with the image of the scene;
obtain a depth map for at least a portion of the scene;
perform, using the one or more processors, one or more image signal processing functions on the image of the scene;
determine an alpha mask for the image of content by comparing depth values for each pixel in the image of content to depth vales for pixels in the processed image of the scene based on the depth map for at least the portion of the scene;
generate a display image by combining, using the one or more processors, the image of the content with the processed image of the scene based at least in part on the alpha mask; and
display, on the display, the display image.

16. The non-transitory computer-readable medium of claim 15 , wherein displaying the display image is performed within 20 milliseconds of capturing the image of the scene.

17. The non-transitory computer-readable medium of claim 15 , wherein the image of the scene is not transmitted via the communications interface, and neither the image of the scene, the processed image of the scene, nor the display image is received via the communications interface.

18. The non-transitory computer-readable medium of claim 15 , wherein the image of the scene is not transmitted via the communications interface.

Assignees

• Apple Inc

Inventors

• Fortin-Deschênes Simon
• Chapdelaine-Couture Vincent
• Côté Yan
• Ghannoum Anthony

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title