Low latency chroma keying embedded in a head-mounted display for mixed reality

Patent metadata

Field	Value
Publication number	US-11778154-B2
Application number	US-201816637087-A
Country	US
Kind code	B2
Filing date	Aug 8, 2018
Priority date	Aug 9, 2017
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

A Head-Mounted Display with camera sensors to perform chroma keying in a mixed reality context is presented. Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content.

First claim

The invention claimed is:

1. A method comprising:
at a head-mounted device including an image sensor, a display, and one or more processors:
capturing, via the image sensor, a live image of a real environment;
performing chromatic aberration correction on the live image, using the one or more processors, to generate a processed image;
obtaining, using the one or more processors, content to be rendered in association with the processed image of the real environment;
detecting, using the one or more processors, a first portion of the processed image of the real environment within a color range;
obtaining, using the one or more processors, a keying mask based on the first portion of the processed image of the real environment;
filtering, using the one or more processors, the keying mask to generate a filtered keying mask corresponding to a second portion of the processed image of the real environment;
modifying, using the one or more processors, the second portion of the processed image of the real environment with the content based on the filtered keying mask, in order to generate a display image; and
displaying, on the display, the display image.

2. The method of claim 1 , wherein a latency between capturing the live image of the real environment and displaying the display image is less than a frame at a frame rate of at least 75 frames per second.

3. The method of claim 1 , wherein the head-mounted device includes a communications interface, and obtaining the content includes receiving the content via the communications interface.

4. The method of claim 3 , wherein the live image of the real environment is not transmitted via the communications interface.

5. The method of claim 3 , wherein neither the live image of the real environment nor the display image is received via the communications interface.

6. The method of claim 1 , wherein the head-mounted device includes one or more positional tracking sensors generating positional tracking data, wherein the content is based on the positional tracking data.

7. The method of claim 6 , wherein the positional tracking sensors include one or more infrared sensors.

8. The method of claim 1 , further comprising, before detecting the first portion of the processed image of the real environment within the color range, performing, using the one or more processors, image processing of the live image of the real environment.

9. The method of claim 8 , wherein the image processing includes at least one of debayering or color format conversion.

10. The method of claim 1 , wherein filtering the keying mask includes performing noise reduction of the keying mask.

11. The method of claim 10 , wherein the noise reduction includes at least one of a morphological filter, blob removal, or edge smoothing.

12. The method of claim 1 , wherein generating the display image includes compositing the processed image of the real environment and the content based on the filtered keying mask.

13. The method of claim 1 , wherein generating the display image includes replacing the second portion of the processed image of the real environment with the content.

14. The method of claim 1 , wherein the keying mask indicates a subset of pixels of the processed image of the real environment, and wherein the subset of pixels corresponds to the second portion of the processed image of the real environment.

15. The method of claim 1 , wherein the keying mask is a chroma keying mask.

16. A head-mounted device comprising:
an image sensor to capture a live image of a real environment;
one or more processors to:
perform chromatic aberration correction on the live image to generate a processed image;
obtain content to be rendered in association with the processed image of the real environment;
detect a first portion of the processed image of the real environment within a color range;
obtain a keying mask based on the first portion of the processed image of the real environment;
filter the keying mask to generate a filtered keying mask corresponding to a second portion of the processed image of the real environment;
modify the second portion of the processed image of the real environment with the content based on the filtered keying mask, in order to generate a display image; and
a display to display the display image.

17. The head-mounted device of claim 16 , wherein a latency between capturing the processed image of the real environment and displaying the display image is less than a frame at a frame rate of at least 75 frames per second.

18. The head-mounted device of claim 16 , further comprising a communications interface, wherein the head-mounted device receives the content via the communications interface.

19. The head-mounted device of claim 18 , wherein the processed image of the real environment is not transmitted via the communications interface.

20. The head-mounted device of claim 16 , further comprising one or more positional tracking sensors generating positional tracking data, wherein the content is based on the positional tracking data.

21. The head-mounted device of claim 20 , wherein the positional tracking sensors include one or more infrared sensors.

22. A non-transitory computer-readable medium having instructions encoded thereon which, when executed by one or more processors of a head-mounted device including an image sensor and a display, cause the head-mounted device to:
capture, via the image sensor, a live image of a real environment;
perform chromatic aberration correction on the live image, using the one or more processors, to generate a processed image;
obtain, using the one or more processors, content to be rendered in association with the processed image of the real environment;
detect, using the one or more processors, a first portion of the processed image of the real environment within a color range;
obtain, using the one or more processors, a keying mask based on the first portion of the processed image of the real environment;
filter, using the one or more processors, the keying mask to generate a filtered keying mask corresponding to a second portion of the processed image of the real environment;
modify, using the one or more processors, the second portion of the processed image of the real environment with the content based on the filtered keying mask, in order to generate a display image; and
display, on the display, the display image.

Assignees

• Apple Inc

Inventors

• Chapdelaine-Couture Vincent
• Ghannoum Anthony
• Cote Yan
• Lustigman Irving
• Chenier Marc-Andre
• Fortin-Deschenes Simon
• Nepveu Bertrand

Classifications

• H04N9/75 Primary

  Chroma key · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N5/265

  Mixing · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N5/2224

  related to virtual studio applications · CPC title