Low latency chroma keying embedded in a head-mounted display for mixed reality

Patent metadata

Field	Value
Publication number	US-2020252593-A1
Application number	US-201816637087-A
Country	US
Kind code	A1
Filing date	Aug 8, 2018
Priority date	Aug 9, 2017
Publication date	Aug 6, 2020
Grant date	—

Abstract

A Head-Mounted Display with camera sensors to perform chroma keying in a mixed reality context is presented. Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content.

First claim

1 - 3 . (canceled)

4 . A method comprising:
at a head-mounted device including an image sensor, a display, and one or more processors: capturing, via the image sensor, an image of a real environment; obtaining content to be rendered in association with the image of the real environment; detecting, using the one or more processors, a first portion of the image of the real environment within a color range; generating, using the one or more processors based on the first portion of the image of the real environment, a keying mask corresponding to a second portion of the image of the real environment; generating, using the one or more processors, a display image based on the image of the real environment, the content, and the keying mask; and displaying, on the display, the display image.

5 . The method of claim 4 , wherein a latency between capturing the image of the real environment and displaying the display image is less than a frame at a frame rate of at least 75 frames per second.

6 . The method of claim 4 , wherein the head-mounted device includes a communications interface and obtaining the content includes receiving the content via the communications interface.

7 . The method of claim 6 , wherein the image of the real environment is not transmitted via the communications interface.

8 . The method of claim 6 , wherein neither the image of the real environment nor the display image is received via the communications interface.

9 . The method of claim 4 , wherein the head-mounted device includes one or more positional tracking sensors generating positional tracking data, wherein the content is based on the positional tracking data.

10 . The method of claim 9 , wherein the positional tracking sensors include one or more infrared sensors.

11 . The method of claim 4 , further comprising, before detecting the first portion of the image of the real environment within the color range, performing, using the one or more processors, image processing of the image of the real environment.

12 . The method of claim 11 , wherein the image processing includes at least one of debayering, chromatic aberration correction, or color format conversion.

13 . The method of claim 4 , wherein generating the keying mask corresponding to the second portion of the image of the real environment includes performing noise reduction of a preliminary mask corresponding to the first portion of the image of the real environment.

14 . The method of claim 13 , wherein the noise reduction includes at least one of a morphological filter, blob removal, or edge smoothing.

15 . The method of claim 4 , wherein generating the display image includes compositing the image of the real environment and the content based on the keying mask.

16 . The method of claim 4 , wherein generating the display image includes replacing the second portion of the image of the real environment with the content.

17 . A head-mounted device comprising:
an image sensor to capture an image of a real environment; a processor to:
obtain content to be rendered in association with the image of the real environment;
detect a first portion of the image of the real environment within a color range;
generate, based on the first portion of the image of the real environment, a keying mask corresponding to a second portion of the image of the real environment; and
generate a display image based on the image of the real environment, the content, and the keying mask; and
a display to display the display image.

18 . The head-mounted device of claim 17 , wherein a latency between capturing the image of the real environment and displaying the display image is less than a frame at a frame rate of at least 75 frames per second.

19 . The head-mounted device of claim 17 , further comprising a communications interface, wherein the head-mounted device receives the content via the communications interface.

20 . The head-mounted device of claim 19 , wherein the image of the real environment is not transmitted via the communications interface.

21 . The head-mounted device of claim 17 , further comprising one or more positional tracking sensors generating positional tracking data, wherein the content is based on the positional tracking data.

22 . The head-mounted device of claim 21 , wherein the positional tracking sensors include one or more infrared sensors.

23 . A non-transitory computer-readable medium having instructions encoded thereon which, when executed by one or more processors of a head-mounted device including an image sensor and a display, cause the head-mounted device to:
capture, via the image sensor, an image of a real environment; obtain content to be rendered in association with the image of the real environment; detect, using the one or more processors, a first portion of the image of the real environment within a color range; generate, using the one or more processors based on the first portion of the image of the real environment, a keying mask corresponding to a second portion of the image of the real environment; generate, using the one or more processors, a display image based on the image of the real environment, the content, and the keying mask; and display, on the display, the display image.

Assignees

• Apple Inc

Inventors

• Chapdelaine-Couture Vincent
• Ghannoum Anthony
• Cote Yan
• Lustigman Irving
• Chenier Marc-Andre
• Fortin-Deschenes Simon
• Nepveu Bertrand

Classifications

• H04N9/75 Primary

  Chroma key · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N5/265

  Mixing · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N5/2224

  related to virtual studio applications · CPC title